2023-04-07 01:53:36,010 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d 2023-04-07 01:53:36,024 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-07 01:53:36,041 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-07 01:53:36,042 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503, deleteOnExit=true 2023-04-07 01:53:36,042 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-07 01:53:36,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/test.cache.data in system properties and HBase conf 2023-04-07 01:53:36,044 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/hadoop.tmp.dir in system properties and HBase conf 2023-04-07 01:53:36,045 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/hadoop.log.dir in system properties and HBase conf 2023-04-07 01:53:36,045 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-07 01:53:36,046 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-07 01:53:36,046 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-07 01:53:36,176 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-07 01:53:36,608 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-07 01:53:36,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-07 01:53:36,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-07 01:53:36,615 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-07 01:53:36,615 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-07 01:53:36,616 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-07 01:53:36,616 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-07 01:53:36,617 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-07 01:53:36,617 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-07 01:53:36,618 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-07 01:53:36,618 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/nfs.dump.dir in system properties and HBase conf 2023-04-07 01:53:36,618 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/java.io.tmpdir in system properties and HBase conf 2023-04-07 01:53:36,619 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-07 01:53:36,619 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-07 01:53:36,620 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-07 01:53:37,220 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-07 01:53:37,225 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-07 01:53:37,517 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-07 01:53:37,684 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-07 01:53:37,708 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-07 01:53:37,756 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-07 01:53:37,790 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/java.io.tmpdir/Jetty_localhost_36811_hdfs____4f469r/webapp 2023-04-07 01:53:37,934 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36811 2023-04-07 01:53:37,950 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-07 01:53:37,950 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-07 01:53:38,539 WARN [Listener at localhost/45891] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-07 01:53:38,673 WARN [Listener at localhost/45891] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-07 01:53:38,693 WARN [Listener at localhost/45891] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-07 01:53:38,701 INFO [Listener at localhost/45891] log.Slf4jLog(67): jetty-6.1.26 2023-04-07 01:53:38,705 INFO [Listener at localhost/45891] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/java.io.tmpdir/Jetty_localhost_37711_datanode____lxp3bu/webapp 2023-04-07 01:53:38,806 INFO [Listener at localhost/45891] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37711 2023-04-07 01:53:39,235 WARN [Listener at localhost/45521] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-07 01:53:39,750 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd0fdc3b5ffc2b40: Processing first storage report for DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d from datanode 46738926-ddcb-4327-bfe7-1a0a7e98fd71 2023-04-07 01:53:39,752 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd0fdc3b5ffc2b40: from storage DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d node DatanodeRegistration(127.0.0.1:44933, datanodeUuid=46738926-ddcb-4327-bfe7-1a0a7e98fd71, infoPort=39511, infoSecurePort=0, ipcPort=45521, storageInfo=lv=-57;cid=testClusterID;nsid=1989836411;c=1680832417299), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-07 01:53:39,753 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd0fdc3b5ffc2b40: Processing first storage report for DS-f127148c-c161-4bf3-b0c2-30304b33c9bf from datanode 46738926-ddcb-4327-bfe7-1a0a7e98fd71 2023-04-07 01:53:39,753 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd0fdc3b5ffc2b40: from storage DS-f127148c-c161-4bf3-b0c2-30304b33c9bf node DatanodeRegistration(127.0.0.1:44933, datanodeUuid=46738926-ddcb-4327-bfe7-1a0a7e98fd71, infoPort=39511, infoSecurePort=0, ipcPort=45521, storageInfo=lv=-57;cid=testClusterID;nsid=1989836411;c=1680832417299), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-07 01:53:39,763 DEBUG [Listener at localhost/45521] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d 2023-04-07 01:53:39,827 INFO [Listener at localhost/45521] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503/zookeeper_0, clientPort=61971, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-07 01:53:39,843 INFO [Listener at localhost/45521] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61971 2023-04-07 01:53:39,853 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:39,855 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:40,520 INFO [Listener at localhost/45521] util.FSUtils(471): Created version file at hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874 with version=8 2023-04-07 01:53:40,521 INFO [Listener at localhost/45521] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/hbase-staging 2023-04-07 01:53:40,842 INFO [Listener at localhost/45521] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-07 01:53:41,341 INFO [Listener at localhost/45521] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-07 01:53:41,372 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,373 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,373 INFO [Listener at localhost/45521] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-07 01:53:41,373 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,373 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-07 01:53:41,518 INFO [Listener at localhost/45521] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-07 01:53:41,592 DEBUG [Listener at localhost/45521] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-07 01:53:41,697 INFO [Listener at localhost/45521] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34673 2023-04-07 01:53:41,708 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:41,710 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:41,733 INFO [Listener at localhost/45521] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34673 connecting to ZooKeeper ensemble=127.0.0.1:61971 2023-04-07 01:53:41,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:346730x0, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-07 01:53:41,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34673-0x1015c5341aa0000 connected 2023-04-07 01:53:41,803 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 01:53:41,804 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 01:53:41,807 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-07 01:53:41,816 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34673 2023-04-07 01:53:41,816 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34673 2023-04-07 01:53:41,817 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34673 2023-04-07 01:53:41,817 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34673 2023-04-07 01:53:41,818 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34673 2023-04-07 01:53:41,824 INFO [Listener at localhost/45521] master.HMaster(444): hbase.rootdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874, hbase.cluster.distributed=false 2023-04-07 01:53:41,899 INFO [Listener at localhost/45521] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-07 01:53:41,899 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,899 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,900 INFO [Listener at localhost/45521] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-07 01:53:41,900 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 01:53:41,900 INFO [Listener at localhost/45521] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-07 01:53:41,906 INFO [Listener at localhost/45521] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-07 01:53:41,908 INFO [Listener at localhost/45521] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44961 2023-04-07 01:53:41,911 INFO [Listener at localhost/45521] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-07 01:53:41,917 DEBUG [Listener at localhost/45521] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-07 01:53:41,917 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:41,920 INFO [Listener at localhost/45521] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:41,922 INFO [Listener at localhost/45521] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44961 connecting to ZooKeeper ensemble=127.0.0.1:61971 2023-04-07 01:53:41,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:449610x0, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-07 01:53:41,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44961-0x1015c5341aa0001 connected 2023-04-07 01:53:41,927 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 01:53:41,928 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 01:53:41,929 DEBUG [Listener at localhost/45521] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-07 01:53:41,930 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44961 2023-04-07 01:53:41,930 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44961 2023-04-07 01:53:41,930 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44961 2023-04-07 01:53:41,931 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44961 2023-04-07 01:53:41,931 DEBUG [Listener at localhost/45521] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44961 2023-04-07 01:53:41,933 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:41,946 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-07 01:53:41,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:41,965 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-07 01:53:41,965 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-07 01:53:41,965 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:41,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-07 01:53:41,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-07 01:53:41,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34673,1680832420667 from backup master directory 2023-04-07 01:53:41,971 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:41,971 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-07 01:53:41,971 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-07 01:53:41,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:41,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-07 01:53:41,976 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-07 01:53:42,460 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/hbase.id with ID: 1be2a76e-f104-434a-b383-816fceb52cd0 2023-04-07 01:53:42,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 01:53:42,519 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:42,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x73bb993b to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:42,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38897a02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:43,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:53:43,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-07 01:53:43,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-07 01:53:43,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-07 01:53:43,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:43,043 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:43,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 01:53:43,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store-tmp 2023-04-07 01:53:43,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:43,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-07 01:53:43,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:53:43,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:53:43,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-07 01:53:43,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:53:43,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:53:43,522 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 01:53:43,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/WALs/jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:43,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34673%2C1680832420667, suffix=, logDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/WALs/jenkins-hbase4.apache.org,34673,1680832420667, archiveDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/oldWALs, maxLogs=10 2023-04-07 01:53:43,603 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK] 2023-04-07 01:53:43,611 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:43,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/WALs/jenkins-hbase4.apache.org,34673,1680832420667/jenkins-hbase4.apache.org%2C34673%2C1680832420667.1680832423559 2023-04-07 01:53:43,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK]] 2023-04-07 01:53:43,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:53:43,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:43,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,674 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,740 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,747 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-07 01:53:43,774 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-07 01:53:43,786 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:43,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 01:53:43,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:53:43,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64829743, jitterRate=-0.033961549401283264}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-07 01:53:43,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 01:53:43,818 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-07 01:53:43,836 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-07 01:53:43,837 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-07 01:53:43,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-07 01:53:43,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-07 01:53:43,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-07 01:53:43,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-07 01:53:43,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-07 01:53:43,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-07 01:53:43,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-07 01:53:43,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-07 01:53:43,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-07 01:53:43,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-07 01:53:43,940 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-07 01:53:43,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:43,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-07 01:53:43,946 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-07 01:53:43,959 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-07 01:53:43,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-07 01:53:43,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-07 01:53:43,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:43,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34673,1680832420667, sessionid=0x1015c5341aa0000, setting cluster-up flag (Was=false) 2023-04-07 01:53:43,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:43,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-07 01:53:43,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:43,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:43,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-07 01:53:44,000 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:44,003 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.hbase-snapshot/.tmp 2023-04-07 01:53:44,036 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(952): ClusterId : 1be2a76e-f104-434a-b383-816fceb52cd0 2023-04-07 01:53:44,041 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-07 01:53:44,047 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-07 01:53:44,047 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-07 01:53:44,051 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-07 01:53:44,052 DEBUG [RS:0;jenkins-hbase4:44961] zookeeper.ReadOnlyZKClient(139): Connect 0x328b66e1 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:44,058 DEBUG [RS:0;jenkins-hbase4:44961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c4c8ec0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:44,058 DEBUG [RS:0;jenkins-hbase4:44961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38730d91, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-07 01:53:44,091 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:44961 2023-04-07 01:53:44,096 INFO [RS:0;jenkins-hbase4:44961] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-07 01:53:44,096 INFO [RS:0;jenkins-hbase4:44961] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-07 01:53:44,096 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1023): About to register with Master. 2023-04-07 01:53:44,100 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34673,1680832420667 with isa=jenkins-hbase4.apache.org/172.31.14.131:44961, startcode=1680832421898 2023-04-07 01:53:44,118 DEBUG [RS:0;jenkins-hbase4:44961] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-07 01:53:44,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-07 01:53:44,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-07 01:53:44,147 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680832454148 2023-04-07 01:53:44,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-07 01:53:44,154 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-07 01:53:44,154 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-07 01:53:44,159 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-07 01:53:44,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-07 01:53:44,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-07 01:53:44,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-07 01:53:44,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-07 01:53:44,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-07 01:53:44,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-07 01:53:44,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-07 01:53:44,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-07 01:53:44,180 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46371, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-07 01:53:44,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-07 01:53:44,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-07 01:53:44,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680832424183,5,FailOnTimeoutGroup] 2023-04-07 01:53:44,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680832424184,5,FailOnTimeoutGroup] 2023-04-07 01:53:44,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-07 01:53:44,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,187 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,208 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-07 01:53:44,209 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-07 01:53:44,210 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874 2023-04-07 01:53:44,223 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874 2023-04-07 01:53:44,223 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:44,223 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45891 2023-04-07 01:53:44,223 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-07 01:53:44,226 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-07 01:53:44,229 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/info 2023-04-07 01:53:44,230 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-07 01:53:44,231 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,231 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-07 01:53:44,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 01:53:44,233 DEBUG [RS:0;jenkins-hbase4:44961] zookeeper.ZKUtil(162): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,233 WARN [RS:0;jenkins-hbase4:44961] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-07 01:53:44,235 INFO [RS:0;jenkins-hbase4:44961] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 01:53:44,235 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/WALs/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,235 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/rep_barrier 2023-04-07 01:53:44,237 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-07 01:53:44,238 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44961,1680832421898] 2023-04-07 01:53:44,238 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-07 01:53:44,243 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/table 2023-04-07 01:53:44,244 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-07 01:53:44,245 DEBUG [RS:0;jenkins-hbase4:44961] zookeeper.ZKUtil(162): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,246 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,248 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740 2023-04-07 01:53:44,250 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740 2023-04-07 01:53:44,254 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:53:44,257 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-07 01:53:44,258 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-07 01:53:44,261 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:53:44,262 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71666536, jitterRate=0.06791460514068604}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:53:44,262 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-07 01:53:44,263 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-07 01:53:44,263 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-07 01:53:44,263 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-07 01:53:44,263 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-07 01:53:44,263 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-07 01:53:44,263 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-07 01:53:44,264 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-07 01:53:44,268 INFO [RS:0;jenkins-hbase4:44961] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-07 01:53:44,269 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-07 01:53:44,269 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-07 01:53:44,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-07 01:53:44,291 INFO [RS:0;jenkins-hbase4:44961] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-07 01:53:44,293 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-07 01:53:44,296 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-07 01:53:44,296 INFO [RS:0;jenkins-hbase4:44961] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-07 01:53:44,297 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,298 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-07 01:53:44,305 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,306 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,306 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,306 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,306 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,307 DEBUG [RS:0;jenkins-hbase4:44961] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 01:53:44,309 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,309 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,309 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,325 INFO [RS:0;jenkins-hbase4:44961] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-07 01:53:44,327 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44961,1680832421898-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,344 INFO [RS:0;jenkins-hbase4:44961] regionserver.Replication(203): jenkins-hbase4.apache.org,44961,1680832421898 started 2023-04-07 01:53:44,344 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44961,1680832421898, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44961, sessionid=0x1015c5341aa0001 2023-04-07 01:53:44,344 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-07 01:53:44,344 DEBUG [RS:0;jenkins-hbase4:44961] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,344 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44961,1680832421898' 2023-04-07 01:53:44,344 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44961,1680832421898' 2023-04-07 01:53:44,345 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-07 01:53:44,346 DEBUG [RS:0;jenkins-hbase4:44961] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-07 01:53:44,346 DEBUG [RS:0;jenkins-hbase4:44961] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-07 01:53:44,347 INFO [RS:0;jenkins-hbase4:44961] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-07 01:53:44,347 INFO [RS:0;jenkins-hbase4:44961] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-07 01:53:44,448 DEBUG [jenkins-hbase4:34673] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-07 01:53:44,450 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44961,1680832421898, state=OPENING 2023-04-07 01:53:44,458 INFO [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44961%2C1680832421898, suffix=, logDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/WALs/jenkins-hbase4.apache.org,44961,1680832421898, archiveDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/oldWALs, maxLogs=32 2023-04-07 01:53:44,460 DEBUG [PEWorker-5] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-07 01:53:44,461 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:44,462 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-07 01:53:44,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:53:44,479 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK] 2023-04-07 01:53:44,483 INFO [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/WALs/jenkins-hbase4.apache.org,44961,1680832421898/jenkins-hbase4.apache.org%2C44961%2C1680832421898.1680832424462 2023-04-07 01:53:44,483 DEBUG [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK]] 2023-04-07 01:53:44,649 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:44,651 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-07 01:53:44,654 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-07 01:53:44,666 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-07 01:53:44,667 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 01:53:44,670 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44961%2C1680832421898.meta, suffix=.meta, logDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/WALs/jenkins-hbase4.apache.org,44961,1680832421898, archiveDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/oldWALs, maxLogs=32 2023-04-07 01:53:44,689 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK] 2023-04-07 01:53:44,693 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/WALs/jenkins-hbase4.apache.org,44961,1680832421898/jenkins-hbase4.apache.org%2C44961%2C1680832421898.meta.1680832424672.meta 2023-04-07 01:53:44,694 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44933,DS-cd7ab8fd-a593-4ea1-b658-5f90ae6c709d,DISK]] 2023-04-07 01:53:44,694 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:53:44,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-07 01:53:44,713 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-07 01:53:44,718 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-07 01:53:44,723 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-07 01:53:44,723 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:44,724 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-07 01:53:44,724 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-07 01:53:44,727 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-07 01:53:44,729 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/info 2023-04-07 01:53:44,729 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/info 2023-04-07 01:53:44,730 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-07 01:53:44,731 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,731 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-07 01:53:44,733 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/rep_barrier 2023-04-07 01:53:44,733 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/rep_barrier 2023-04-07 01:53:44,733 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-07 01:53:44,734 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,734 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-07 01:53:44,736 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/table 2023-04-07 01:53:44,736 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/table 2023-04-07 01:53:44,737 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-07 01:53:44,737 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:44,740 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740 2023-04-07 01:53:44,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740 2023-04-07 01:53:44,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:53:44,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-07 01:53:44,749 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59220380, jitterRate=-0.11754757165908813}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:53:44,750 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-07 01:53:44,759 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680832424642 2023-04-07 01:53:44,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-07 01:53:44,776 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-07 01:53:44,778 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44961,1680832421898, state=OPEN 2023-04-07 01:53:44,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-07 01:53:44,782 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-07 01:53:44,786 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-07 01:53:44,786 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44961,1680832421898 in 317 msec 2023-04-07 01:53:44,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-07 01:53:44,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 510 msec 2023-04-07 01:53:44,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 745 msec 2023-04-07 01:53:44,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680832424796, completionTime=-1 2023-04-07 01:53:44,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-07 01:53:44,797 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-07 01:53:44,863 DEBUG [hconnection-0x33c18644-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:44,867 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:44,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-07 01:53:44,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680832484884 2023-04-07 01:53:44,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680832544884 2023-04-07 01:53:44,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-04-07 01:53:44,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34673,1680832420667-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34673,1680832420667-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34673,1680832420667-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34673, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-07 01:53:44,916 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-07 01:53:44,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-07 01:53:44,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-07 01:53:44,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-07 01:53:44,941 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:53:44,943 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:53:44,967 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:44,970 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810 empty. 2023-04-07 01:53:44,971 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:44,971 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-07 01:53:45,394 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-07 01:53:45,396 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => fd36335cf6a580f3ebcc2199ad95d810, NAME => 'hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing fd36335cf6a580f3ebcc2199ad95d810, disabling compactions & flushes 2023-04-07 01:53:45,810 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. after waiting 0 ms 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:45,810 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:45,810 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for fd36335cf6a580f3ebcc2199ad95d810: 2023-04-07 01:53:45,815 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:53:45,830 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680832425818"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832425818"}]},"ts":"1680832425818"} 2023-04-07 01:53:45,854 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:53:45,856 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:53:45,860 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832425856"}]},"ts":"1680832425856"} 2023-04-07 01:53:45,864 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-07 01:53:45,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fd36335cf6a580f3ebcc2199ad95d810, ASSIGN}] 2023-04-07 01:53:45,877 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fd36335cf6a580f3ebcc2199ad95d810, ASSIGN 2023-04-07 01:53:45,878 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=fd36335cf6a580f3ebcc2199ad95d810, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:53:46,030 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fd36335cf6a580f3ebcc2199ad95d810, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:46,030 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680832426029"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832426029"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832426029"}]},"ts":"1680832426029"} 2023-04-07 01:53:46,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure fd36335cf6a580f3ebcc2199ad95d810, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:53:46,194 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:46,195 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fd36335cf6a580f3ebcc2199ad95d810, NAME => 'hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:53:46,196 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,196 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:46,196 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,196 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,198 INFO [StoreOpener-fd36335cf6a580f3ebcc2199ad95d810-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,200 DEBUG [StoreOpener-fd36335cf6a580f3ebcc2199ad95d810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/info 2023-04-07 01:53:46,200 DEBUG [StoreOpener-fd36335cf6a580f3ebcc2199ad95d810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/info 2023-04-07 01:53:46,201 INFO [StoreOpener-fd36335cf6a580f3ebcc2199ad95d810-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fd36335cf6a580f3ebcc2199ad95d810 columnFamilyName info 2023-04-07 01:53:46,202 INFO [StoreOpener-fd36335cf6a580f3ebcc2199ad95d810-1] regionserver.HStore(310): Store=fd36335cf6a580f3ebcc2199ad95d810/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:46,204 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,204 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,210 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:53:46,213 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:53:46,214 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fd36335cf6a580f3ebcc2199ad95d810; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63820811, jitterRate=-0.04899580776691437}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-07 01:53:46,214 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fd36335cf6a580f3ebcc2199ad95d810: 2023-04-07 01:53:46,216 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810., pid=6, masterSystemTime=1680832426188 2023-04-07 01:53:46,220 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:46,220 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:53:46,222 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fd36335cf6a580f3ebcc2199ad95d810, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:46,222 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680832426221"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832426221"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832426221"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832426221"}]},"ts":"1680832426221"} 2023-04-07 01:53:46,231 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-07 01:53:46,231 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure fd36335cf6a580f3ebcc2199ad95d810, server=jenkins-hbase4.apache.org,44961,1680832421898 in 193 msec 2023-04-07 01:53:46,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-07 01:53:46,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=fd36335cf6a580f3ebcc2199ad95d810, ASSIGN in 357 msec 2023-04-07 01:53:46,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:53:46,237 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832426237"}]},"ts":"1680832426237"} 2023-04-07 01:53:46,240 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-07 01:53:46,244 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:53:46,245 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-07 01:53:46,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-07 01:53:46,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:53:46,247 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3160 sec 2023-04-07 01:53:46,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-07 01:53:46,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-07 01:53:46,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-04-07 01:53:46,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-07 01:53:46,328 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-07 01:53:46,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-07 01:53:46,344 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-07 01:53:46,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-07 01:53:46,347 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.375sec 2023-04-07 01:53:46,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-07 01:53:46,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-07 01:53:46,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-07 01:53:46,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34673,1680832420667-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-07 01:53:46,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34673,1680832420667-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-07 01:53:46,362 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-07 01:53:46,443 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x0f3002be to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:46,448 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e324982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:46,465 DEBUG [hconnection-0x10a3ae0a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:46,477 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:46,488 INFO [Listener at localhost/45521] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:53:46,514 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=169, AvailableMemoryMB=3493 2023-04-07 01:53:46,531 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:53:46,535 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:53:46,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:53:46,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:53:46,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:53:46,551 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:53:46,554 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:53:46,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-07 01:53:46,557 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,558 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c empty. 2023-04-07 01:53:46,560 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,560 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:53:46,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 01:53:46,576 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:53:46,577 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e2fe8c1647b59c84578d50aa3cf4039c, NAME => 'TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:53:46,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:46,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e2fe8c1647b59c84578d50aa3cf4039c, disabling compactions & flushes 2023-04-07 01:53:46,590 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,590 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,590 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. after waiting 0 ms 2023-04-07 01:53:46,590 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,590 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,590 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:46,594 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:53:46,595 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832426595"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832426595"}]},"ts":"1680832426595"} 2023-04-07 01:53:46,598 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:53:46,599 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:53:46,600 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832426599"}]},"ts":"1680832426599"} 2023-04-07 01:53:46,602 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:53:46,605 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, ASSIGN}] 2023-04-07 01:53:46,607 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, ASSIGN 2023-04-07 01:53:46,609 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:53:46,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 01:53:46,760 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e2fe8c1647b59c84578d50aa3cf4039c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:46,760 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832426760"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832426760"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832426760"}]},"ts":"1680832426760"} 2023-04-07 01:53:46,764 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:53:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 01:53:46,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e2fe8c1647b59c84578d50aa3cf4039c, NAME => 'TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,936 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,939 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A 2023-04-07 01:53:46,939 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A 2023-04-07 01:53:46,941 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:53:46,941 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e2fe8c1647b59c84578d50aa3cf4039c columnFamilyName A 2023-04-07 01:53:46,943 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(310): Store=e2fe8c1647b59c84578d50aa3cf4039c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:46,944 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,947 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B 2023-04-07 01:53:46,947 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B 2023-04-07 01:53:46,948 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:53:46,948 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e2fe8c1647b59c84578d50aa3cf4039c columnFamilyName B 2023-04-07 01:53:46,950 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(310): Store=e2fe8c1647b59c84578d50aa3cf4039c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:46,950 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,953 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C 2023-04-07 01:53:46,953 DEBUG [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C 2023-04-07 01:53:46,953 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:53:46,953 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e2fe8c1647b59c84578d50aa3cf4039c columnFamilyName C 2023-04-07 01:53:46,954 INFO [StoreOpener-e2fe8c1647b59c84578d50aa3cf4039c-1] regionserver.HStore(310): Store=e2fe8c1647b59c84578d50aa3cf4039c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:53:46,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:53:46,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:46,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:53:46,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e2fe8c1647b59c84578d50aa3cf4039c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68945408, jitterRate=0.02736663818359375}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:53:46,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:46,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., pid=11, masterSystemTime=1680832426917 2023-04-07 01:53:46,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:46,977 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e2fe8c1647b59c84578d50aa3cf4039c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:46,977 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832426977"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832426977"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832426977"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832426977"}]},"ts":"1680832426977"} 2023-04-07 01:53:46,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-07 01:53:46,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 in 216 msec 2023-04-07 01:53:46,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-07 01:53:46,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, ASSIGN in 379 msec 2023-04-07 01:53:46,989 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:53:46,990 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832426990"}]},"ts":"1680832426990"} 2023-04-07 01:53:46,992 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:53:46,997 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:53:46,999 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 449 msec 2023-04-07 01:53:47,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 01:53:47,176 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-07 01:53:47,181 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,186 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,189 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,194 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,197 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:53:47,200 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:53:47,207 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,211 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,212 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,218 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,219 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,226 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,227 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,231 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,231 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,235 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,235 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,239 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,240 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,264 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,265 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,268 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,268 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:53:47,272 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:53:47,275 DEBUG [hconnection-0x1865dd73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,276 DEBUG [hconnection-0x5ad720d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,277 DEBUG [hconnection-0x63093886-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,277 DEBUG [hconnection-0x2db96cc3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,280 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,281 DEBUG [hconnection-0x3e1787eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,282 DEBUG [hconnection-0x635c9341-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,290 DEBUG [hconnection-0x2be07eef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,290 DEBUG [hconnection-0x6d34219d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:53:47,293 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,293 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38402, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,300 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,300 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,301 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,300 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,302 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,302 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,302 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:53:47,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:47,334 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:47,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:47,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:47,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:47,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:47,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:47,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:47,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:47,340 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:47,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:47,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:47,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:47,357 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:47,362 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:47,364 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-07 01:53:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:47,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:47,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:47,387 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:47,387 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:47,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:47,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:47,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:47,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832427444, max timestamp=1680832427488Num uniques 859; 2023-04-07 01:53:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832427444, max timestamp=1680832427472Num uniques 859; 2023-04-07 01:53:47,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:47,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832487470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:47,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832487470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832427444, max timestamp=1680832427472Num uniques 859; 2023-04-07 01:53:47,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:47,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832487472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,516 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/eae1622b2da444fcb3e5c35317841946 2023-04-07 01:53:47,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832487616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832487618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832487647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832487648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832487649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,671 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/65ba28173584436b8112d431e4cebd76 2023-04-07 01:53:47,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832487736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832487736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832487880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832487882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832487882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:47,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832487945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:47,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832487945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,143 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/ed17282226e14781b2d0d2b6316dc6f7 2023-04-07 01:53:48,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/eae1622b2da444fcb3e5c35317841946 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946 2023-04-07 01:53:48,170 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:53:48,174 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 01:53:48,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/65ba28173584436b8112d431e4cebd76 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76 2023-04-07 01:53:48,191 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:53:48,198 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/ed17282226e14781b2d0d2b6316dc6f7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7 2023-04-07 01:53:48,208 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:53:48,211 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e2fe8c1647b59c84578d50aa3cf4039c in 847ms, sequenceid=12, compaction requested=false 2023-04-07 01:53:48,214 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 01:53:48,217 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:48,217 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:48,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:48,233 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1152b627[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:48,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:48,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:48,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:48,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:48,254 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:48,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:48,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:48,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:48,254 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:48,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,259 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,262 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:48,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=147.60 KB heapSize=425.53 KB 2023-04-07 01:53:48,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:48,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:53:48,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:48,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:53:48,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:48,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:48,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:48,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:48,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:48,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832488274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832488299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832488299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832488302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832488302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/4fb4b31cd52e4526b7c1e2953ef1bf83 2023-04-07 01:53:48,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:48,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832488407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:48,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:48,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:48,434 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:48,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:48,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:48,439 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,439 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:48,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:48,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832488433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:48,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832488433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:48,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832488433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832488444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,451 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:48,452 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:48,466 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d49951e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:48,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:48,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,473 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:48,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:48,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:48,476 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:48,483 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:48,483 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:48,483 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:48,483 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:48,483 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:48,488 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,488 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,488 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:48,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:48,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,505 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:48,506 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:48,606 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:48,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:48,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:48,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:48,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:48,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,620 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:48,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832488616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:48,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:48,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:48,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:48,645 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1e08b7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:48,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:48,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:48,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832488651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832488658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832488659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832488659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:48,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:48,664 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:48,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:48,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:48,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:48,664 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:48,668 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,669 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,669 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:48,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:48,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,684 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,685 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:48,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:48,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:48,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:48,793 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:48,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:48,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:48,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,800 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:48,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:48,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:48,809 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,809 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:48,809 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:48,812 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:48,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:48,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22ba535[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:48,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:48,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:48,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:48,829 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:48,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:48,837 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:48,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:48,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:48,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:48,837 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:48,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:48,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:48,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:48,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:48,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:48,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:48,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:48,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:48,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:48,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:48,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/e56339d0e0f447fd9bd088f33d90a12f 2023-04-07 01:53:48,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/a7be4c3f145543b9a99dee42197208cc 2023-04-07 01:53:48,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832488929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/4fb4b31cd52e4526b7c1e2953ef1bf83 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83 2023-04-07 01:53:48,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83, entries=300, sequenceid=52, filesize=18.5 K 2023-04-07 01:53:48,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/e56339d0e0f447fd9bd088f33d90a12f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f 2023-04-07 01:53:48,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:48,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:48,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f, entries=150, sequenceid=52, filesize=11.6 K 2023-04-07 01:53:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:48,965 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:48,966 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:48,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/a7be4c3f145543b9a99dee42197208cc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc 2023-04-07 01:53:48,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,971 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:48,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:48,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:48,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832488964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832488964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:48,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832488965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832488971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:48,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:48,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc, entries=150, sequenceid=52, filesize=11.6 K 2023-04-07 01:53:48,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~445.41 KB/456096, currentSize=26.84 KB/27480 for e2fe8c1647b59c84578d50aa3cf4039c in 712ms, sequenceid=52, compaction requested=false 2023-04-07 01:53:48,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:48,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:48,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:48,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:48,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:48,989 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:48,990 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:48,990 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 01:53:48,990 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:48,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:48,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:49,042 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/e9220123b5c9495aa70f4a9d96024f24 2023-04-07 01:53:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:49,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680832429483, max timestamp=1680832429495Num uniques 859; 2023-04-07 01:53:49,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7642, suffix heap size=117064,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=167 2023-04-07 01:53:49,511 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/d00f989b56ce4d5890382f07281e67ed 2023-04-07 01:53:49,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832489518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832489520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832489523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832489524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832489524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,574 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/aa56f9607bd1437b868f8aa3b348695f 2023-04-07 01:53:49,589 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/e9220123b5c9495aa70f4a9d96024f24 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24 2023-04-07 01:53:49,602 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24, entries=100, sequenceid=59, filesize=9.3 K 2023-04-07 01:53:49,607 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/d00f989b56ce4d5890382f07281e67ed as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed 2023-04-07 01:53:49,621 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed, entries=100, sequenceid=59, filesize=9.3 K 2023-04-07 01:53:49,630 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/aa56f9607bd1437b868f8aa3b348695f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f 2023-04-07 01:53:49,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832489631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832489632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832489632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832489639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832489639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,646 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f, entries=100, sequenceid=59, filesize=9.3 K 2023-04-07 01:53:49,648 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=150.19 KB/153798 for e2fe8c1647b59c84578d50aa3cf4039c in 658ms, sequenceid=59, compaction requested=true 2023-04-07 01:53:49,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:49,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:49,651 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,651 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cc8d9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:49,657 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:49,657 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,659 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:49,660 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,660 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:49,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:49,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:49,670 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:49,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:49,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:49,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:49,670 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:49,672 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,672 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:49,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,672 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:49,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,688 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:49,688 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:49,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:49,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:49,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:49,689 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:49,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,692 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,692 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,692 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:49,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:49,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:49,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:49,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:49,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:49,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:49,830 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:49,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:49,831 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:49,831 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:49,831 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,831 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,838 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,838 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:49,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:49,838 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,838 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:49,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=170.32 KB heapSize=501.98 KB 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:49,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:49,863 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:49,863 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:49,863 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:49,863 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:49,863 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:49,863 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:49,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832489854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832489854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832489854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,877 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832489871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832489855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,885 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ac5c6b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:49,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:49,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:49,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:49,894 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:49,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:49,907 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:49,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:49,907 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:49,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:49,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=36.14 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2acb45dca0c0421380dc580c89bed371 2023-04-07 01:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:49,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:49,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:49,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:49,930 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:49,930 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:49,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:49,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:49,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8e99c3d816664145b295e72ae0aa0f52 2023-04-07 01:53:49,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832489983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832489985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832489988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:49,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832489991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,036 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,040 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,041 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,041 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d6da799[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,080 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,181 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832490186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,189 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832490187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832490191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832490193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832490200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4230ab0c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,232 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,232 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,235 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,235 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,235 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,237 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,252 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,252 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:50,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,270 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,304 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:53:50,369 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-07 01:53:50,370 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-07 01:53:50,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,384 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,387 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,388 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22daa993[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,417 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,417 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,424 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,424 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,432 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/7e047054e7114256bb6674cc4c957392 2023-04-07 01:53:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:50,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2acb45dca0c0421380dc580c89bed371 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371 2023-04-07 01:53:50,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371, entries=150, sequenceid=92, filesize=11.6 K 2023-04-07 01:53:50,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8e99c3d816664145b295e72ae0aa0f52 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52 2023-04-07 01:53:50,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52, entries=150, sequenceid=92, filesize=11.6 K 2023-04-07 01:53:50,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/7e047054e7114256bb6674cc4c957392 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392 2023-04-07 01:53:50,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832490492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832490498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392, entries=150, sequenceid=92, filesize=11.6 K 2023-04-07 01:53:50,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~170.32 KB/174408, heapSize ~501.94 KB/513984, currentSize=6.71 KB/6870 for e2fe8c1647b59c84578d50aa3cf4039c in 661ms, sequenceid=92, compaction requested=true 2023-04-07 01:53:50,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:50,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:50,523 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:50,523 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:50,523 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:50,523 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:50,530 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:50,530 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:50,532 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:50,532 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:50,533 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,533 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,533 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=44.1 K 2023-04-07 01:53:50,533 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=51.0 K 2023-04-07 01:53:50,536 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 65ba28173584436b8112d431e4cebd76, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832427346 2023-04-07 01:53:50,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,541 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting eae1622b2da444fcb3e5c35317841946, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832427346 2023-04-07 01:53:50,541 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e56339d0e0f447fd9bd088f33d90a12f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680832427589 2023-04-07 01:53:50,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,543 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4fb4b31cd52e4526b7c1e2953ef1bf83, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680832427589 2023-04-07 01:53:50,544 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d00f989b56ce4d5890382f07281e67ed, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680832428298 2023-04-07 01:53:50,544 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e9220123b5c9495aa70f4a9d96024f24, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680832428298 2023-04-07 01:53:50,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,545 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8e99c3d816664145b295e72ae0aa0f52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:50,546 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2acb45dca0c0421380dc580c89bed371, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:50,550 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:53:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,554 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,554 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:50,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:50,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:50,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,578 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,578 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,578 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,578 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,580 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4770f6a7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,597 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,597 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,597 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#14 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:50,598 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#13 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:50,600 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,604 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,603 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:50,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2036be2c74db4a41bfa9f6656c62d963 2023-04-07 01:53:50,680 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/88bb288eba544aef9884e215ddc81ab4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/88bb288eba544aef9884e215ddc81ab4 2023-04-07 01:53:50,689 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/72723068a60f44758bbb2c49efc05d7c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/72723068a60f44758bbb2c49efc05d7c 2023-04-07 01:53:50,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/67a80c7d00c74e30b01a0656b3839aa9 2023-04-07 01:53:50,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,722 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into 72723068a60f44758bbb2c49efc05d7c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:50,722 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 88bb288eba544aef9884e215ddc81ab4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:50,723 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:50,723 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=12, startTime=1680832430522; duration=0sec 2023-04-07 01:53:50,723 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:50,723 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=12, startTime=1680832430523; duration=0sec 2023-04-07 01:53:50,724 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:50,724 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:50,724 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,725 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832430669, max timestamp=1680832430717Num uniques 859; 2023-04-07 01:53:50,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:50,728 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832430669, max timestamp=1680832430717Num uniques 859; 2023-04-07 01:53:50,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1680832430669, max timestamp=1680832430714Num uniques 859; 2023-04-07 01:53:50,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8240, suffix heap size=117064,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=180 2023-04-07 01:53:50,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:50,731 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:50,731 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:50,731 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,731 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=44.1 K 2023-04-07 01:53:50,732 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ed17282226e14781b2d0d2b6316dc6f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832427346 2023-04-07 01:53:50,732 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a7be4c3f145543b9a99dee42197208cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680832427589 2023-04-07 01:53:50,737 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,738 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting aa56f9607bd1437b868f8aa3b348695f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680832428298 2023-04-07 01:53:50,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,740 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7e047054e7114256bb6674cc4c957392, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:50,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3751c0bf[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,758 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,765 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,765 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,766 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,769 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,769 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,769 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:50,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,781 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,781 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,783 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#17 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,787 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,787 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,838 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-07 01:53:50,839 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-07 01:53:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832490853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832490855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832490857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:50,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:50,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:50,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:50,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:50,891 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:50,891 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:50,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:50,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,896 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:50,898 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:50,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:50,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:50,901 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,901 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:50,901 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:50,901 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:50,906 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,907 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,907 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe6592[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:50,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:50,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,911 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:50,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:50,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:50,921 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:50,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:50,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:50,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:50,921 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:50,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:50,927 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:50,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:50,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:50,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:50,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:50,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:50,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:50,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:50,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832490964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832490966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:50,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832490967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832491008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832491008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,034 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:51,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:51,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:51,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:51,040 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:51,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:51,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:51,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,047 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:51,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:51,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:51,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:51,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:51,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71fdc2f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:51,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:51,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,060 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:51,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:51,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:51,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:51,071 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:51,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:51,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:51,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:51,071 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:51,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,082 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:51,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:51,085 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:51,085 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:51,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/4623d26d63f740d9adb927a542b69fb4 2023-04-07 01:53:51,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2036be2c74db4a41bfa9f6656c62d963 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963 2023-04-07 01:53:51,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963, entries=150, sequenceid=102, filesize=11.6 K 2023-04-07 01:53:51,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/67a80c7d00c74e30b01a0656b3839aa9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9 2023-04-07 01:53:51,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832491170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832491171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832491171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9, entries=150, sequenceid=102, filesize=11.6 K 2023-04-07 01:53:51,186 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:51,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/4623d26d63f740d9adb927a542b69fb4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4 2023-04-07 01:53:51,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:51,195 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:51,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:51,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:51,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:51,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:51,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:51,218 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,218 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,222 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:51,222 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,226 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471ac7a5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:51,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:51,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4, entries=150, sequenceid=102, filesize=11.6 K 2023-04-07 01:53:51,237 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:51,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=135.12 KB/138366 for e2fe8c1647b59c84578d50aa3cf4039c in 690ms, sequenceid=102, compaction requested=false 2023-04-07 01:53:51,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:51,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,251 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/655d6e697b71490485134306d7d9ce04 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/655d6e697b71490485134306d7d9ce04 2023-04-07 01:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:51,254 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:51,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:51,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:51,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:51,254 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:51,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,258 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,259 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,260 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,262 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 655d6e697b71490485134306d7d9ce04(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:51,262 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:51,262 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=12, startTime=1680832430523; duration=0sec 2023-04-07 01:53:51,264 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:51,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:51,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:51,267 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,268 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:51,268 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:51,368 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:51,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:51,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:51,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:51,376 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:51,376 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:51,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:51,378 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,378 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,380 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,380 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,380 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,380 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:51,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:51,387 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:51,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:51,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:51,388 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=135.12 KB heapSize=387.12 KB 2023-04-07 01:53:51,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:51,388 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:51,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:51,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:51,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:51,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:51,420 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=45.67 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/30788cc88752493f80e109d30ad43596 2023-04-07 01:53:51,470 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ce0ad245e2ca47abbcaf4de691e387fa 2023-04-07 01:53:51,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:51,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:51,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832491498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832491498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832491500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,522 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/ef552b22118e495fb3343a1d0b671de5 2023-04-07 01:53:51,532 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/30788cc88752493f80e109d30ad43596 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596 2023-04-07 01:53:51,541 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596, entries=150, sequenceid=142, filesize=11.8 K 2023-04-07 01:53:51,544 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ce0ad245e2ca47abbcaf4de691e387fa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa 2023-04-07 01:53:51,556 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa, entries=150, sequenceid=142, filesize=11.8 K 2023-04-07 01:53:51,559 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/ef552b22118e495fb3343a1d0b671de5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5 2023-04-07 01:53:51,569 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5, entries=150, sequenceid=142, filesize=11.8 K 2023-04-07 01:53:51,572 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~135.12 KB/138366, heapSize ~387.07 KB/396360, currentSize=46.96 KB/48090 for e2fe8c1647b59c84578d50aa3cf4039c in 184ms, sequenceid=142, compaction requested=true 2023-04-07 01:53:51,572 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:51,572 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:51,589 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,589 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:51,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef7a3c1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:51,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:51,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:51,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:51,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:51,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:51,599 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:51,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:53:51,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:51,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:51,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:51,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:51,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:51,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:51,617 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:51,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:51,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:51,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:51,617 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:51,626 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,626 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,626 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:51,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:51,634 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,634 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:51,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,634 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:51,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/725ea0c3e2a44df8bd1c12b51444a9e4 2023-04-07 01:53:51,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832491667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832431649, max timestamp=1680832431669Num uniques 859; 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832431649, max timestamp=1680832431669Num uniques 859; 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832431649, max timestamp=1680832431669Num uniques 859; 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:51,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:51,735 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:51,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:51,742 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:51,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:51,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:51,745 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,745 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:51,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:51,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:51,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:51,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832491753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832491756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:51,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:51,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:51,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:51,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832491774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:51,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d269b78[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:51,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:51,780 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,783 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:51,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:51,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:51,793 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:51,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:51,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:51,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:51,793 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:51,795 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,795 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,795 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,803 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,803 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:51,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:51,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,804 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:51,804 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:51,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:51,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832491862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832491863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,904 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:51,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:51,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:51,910 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:51,910 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:51,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:51,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:51,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:51,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:51,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:51,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:51,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:51,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1de57[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:51,927 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:51,927 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,928 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:51,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:51,937 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:51,942 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:51,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:51,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:51,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:51,942 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:51,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,945 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,947 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:51,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:51,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:51,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:51,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:51,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:51,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:51,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:51,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:51,954 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:51,955 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:51,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:51,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:51,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:51,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832491979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832492020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832492023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,055 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:52,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:52,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:52,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:52,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:52,064 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:52,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:52,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:52,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832492070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832492073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,079 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:52,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:52,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:52,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:52,095 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:52,097 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/3da1932af7034f309d5e0d1c86763b6c 2023-04-07 01:53:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:52,099 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:52,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15170689[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:52,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:52,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:52,109 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:52,109 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,109 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:52,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,111 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,111 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:52,121 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:52,121 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:52,121 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:52,121 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:52,121 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:52,129 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,129 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,129 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,131 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:52,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:52,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:52,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,143 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:52,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:52,143 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/456ea6e1587e48c28c843d62f947bbc7 2023-04-07 01:53:52,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/725ea0c3e2a44df8bd1c12b51444a9e4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4 2023-04-07 01:53:52,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4, entries=150, sequenceid=154, filesize=11.8 K 2023-04-07 01:53:52,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/3da1932af7034f309d5e0d1c86763b6c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c 2023-04-07 01:53:52,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c, entries=150, sequenceid=154, filesize=11.8 K 2023-04-07 01:53:52,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/456ea6e1587e48c28c843d62f947bbc7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7 2023-04-07 01:53:52,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7, entries=150, sequenceid=154, filesize=11.8 K 2023-04-07 01:53:52,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for e2fe8c1647b59c84578d50aa3cf4039c in 571ms, sequenceid=154, compaction requested=true 2023-04-07 01:53:52,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:52,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:52,185 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:52,185 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:52,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:52,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:52,188 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:52,188 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:52,188 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:52,188 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:52,188 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,188 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,189 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/88bb288eba544aef9884e215ddc81ab4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=46.9 K 2023-04-07 01:53:52,189 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/72723068a60f44758bbb2c49efc05d7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=46.9 K 2023-04-07 01:53:52,189 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 88bb288eba544aef9884e215ddc81ab4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:52,191 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 72723068a60f44758bbb2c49efc05d7c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:52,195 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2036be2c74db4a41bfa9f6656c62d963, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680832430527 2023-04-07 01:53:52,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 67a80c7d00c74e30b01a0656b3839aa9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680832430527 2023-04-07 01:53:52,195 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 30788cc88752493f80e109d30ad43596, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680832430808 2023-04-07 01:53:52,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ce0ad245e2ca47abbcaf4de691e387fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680832430808 2023-04-07 01:53:52,196 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 725ea0c3e2a44df8bd1c12b51444a9e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:52,196 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3da1932af7034f309d5e0d1c86763b6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:52,210 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:52,215 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:52,244 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:52,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:52,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:52,258 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/3e246d43ce9e4d428ea9a05be5f1fc06 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3e246d43ce9e4d428ea9a05be5f1fc06 2023-04-07 01:53:52,258 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/11559caf4382417e9c158cd0257265df as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/11559caf4382417e9c158cd0257265df 2023-04-07 01:53:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:52,264 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:52,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:52,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:52,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:52,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:52,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:52,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:52,290 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-07 01:53:52,292 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into 3e246d43ce9e4d428ea9a05be5f1fc06(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:52,292 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:52,292 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=12, startTime=1680832432185; duration=0sec 2023-04-07 01:53:52,292 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:52,292 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:52,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:52,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:52,294 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:52,294 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:52,294 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 11559caf4382417e9c158cd0257265df(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:52,295 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:52,294 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:52,295 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:52,295 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:52,295 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:52,295 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:52,297 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:52,297 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,297 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/655d6e697b71490485134306d7d9ce04, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=46.9 K 2023-04-07 01:53:52,295 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=12, startTime=1680832432185; duration=0sec 2023-04-07 01:53:52,297 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:52,298 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 655d6e697b71490485134306d7d9ce04, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680832429520 2023-04-07 01:53:52,300 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4623d26d63f740d9adb927a542b69fb4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680832430527 2023-04-07 01:53:52,301 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ef552b22118e495fb3343a1d0b671de5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680832430808 2023-04-07 01:53:52,301 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 456ea6e1587e48c28c843d62f947bbc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:52,331 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:52,364 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/4f63fb0e25db4fbe8bd31a6a802dc153 2023-04-07 01:53:52,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832492377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832492378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832492378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,391 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8a1883e22bef4ae783d76463b9f92ab0 2023-04-07 01:53:52,424 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/e68ac8c11eae47af89d756e10cfcafb3 2023-04-07 01:53:52,432 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/4f63fb0e25db4fbe8bd31a6a802dc153 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153 2023-04-07 01:53:52,439 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153, entries=200, sequenceid=191, filesize=14.1 K 2023-04-07 01:53:52,442 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8a1883e22bef4ae783d76463b9f92ab0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0 2023-04-07 01:53:52,451 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0, entries=150, sequenceid=191, filesize=11.8 K 2023-04-07 01:53:52,454 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/e68ac8c11eae47af89d756e10cfcafb3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3 2023-04-07 01:53:52,464 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3, entries=150, sequenceid=191, filesize=11.8 K 2023-04-07 01:53:52,469 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e2fe8c1647b59c84578d50aa3cf4039c in 179ms, sequenceid=191, compaction requested=false 2023-04-07 01:53:52,469 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:52,469 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:52,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:52,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd4641[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:52,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:52,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:52,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:52,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:52,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:52,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:52,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:53:52,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:52,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:52,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:52,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:52,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:52,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:52,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:52,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:52,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:52,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:52,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:52,497 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,497 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,498 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,508 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:52,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:52,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:52,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:52,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832492561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:52,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:52,614 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:52,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:52,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:52,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:52,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:52,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:52,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:52,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:52,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:52,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:52,625 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f89eab5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:52,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:52,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:52,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:52,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:52,637 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:52,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:52,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:52,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:52,637 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:52,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,645 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:52,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:52,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:52,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:52,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832492666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:52,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:52,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:52,759 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:52,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:52,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:52,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,763 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,763 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:52,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:52,767 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,767 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:52,768 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:52,768 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:52,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,775 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/6c87601642794c8283960d1aedf4e793 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/6c87601642794c8283960d1aedf4e793 2023-04-07 01:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1550cc75[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:52,776 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:52,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:52,780 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:52,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:52,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:52,789 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:52,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:52,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:52,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:52,789 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:52,791 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 6c87601642794c8283960d1aedf4e793(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:52,797 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:52,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,797 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=12, startTime=1680832432185; duration=0sec 2023-04-07 01:53:52,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,798 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,807 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:52,807 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:52,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832492869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832492888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832492890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:52,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:52,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:52,913 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:52,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:52,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:52,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:52,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:52,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:52,920 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,920 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:52,920 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:52,920 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:52,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/73566c1c2f204e5dbb4776cd1b5031fd 2023-04-07 01:53:52,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38dea318[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:52,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:52,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,928 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:52,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:52,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:52,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:52,935 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:52,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:52,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:52,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:52,935 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:52,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:52,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:52,939 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:52,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:52,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:52,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:52,944 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:52,944 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:52,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:52,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:53,045 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:53,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:53,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:53,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:53,049 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:53,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:53,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:53,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:53,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:53,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:53,054 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,054 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:53,054 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:53,054 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:53,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:53,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed750c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:53,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:53,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,060 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:53,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:53,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:53,073 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:53,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:53,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:53,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:53,073 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:53,074 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,074 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,075 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,079 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:53,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:53,079 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:53,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:53,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832493175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,179 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:53,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:53,184 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:53,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:53,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:53,193 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,193 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:53,193 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:53,193 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:53,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e1eb66[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:53,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:53,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,201 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:53,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:53,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,203 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:53,208 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:53,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:53,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:53,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:53,208 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:53,210 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,210 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,210 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,214 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:53,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:53,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,222 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:53,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,222 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:53,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,322 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:53,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:53,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:53,330 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:53,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:53,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:53,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,332 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:53,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:53,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:53,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,337 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:53,337 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:53,337 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:53,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4741a6eb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:53,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:53,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:53,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:53,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/55e4b92a8d574b97ab84583c4ce9221f 2023-04-07 01:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:53,359 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:53,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:53,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:53,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:53,359 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:53,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,369 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:53,374 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:53,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:53,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:53,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:53,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:53,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:53,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/43e7f54812194461a484fa95decfe248 2023-04-07 01:53:53,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/73566c1c2f204e5dbb4776cd1b5031fd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd 2023-04-07 01:53:53,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd, entries=200, sequenceid=203, filesize=14.1 K 2023-04-07 01:53:53,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/55e4b92a8d574b97ab84583c4ce9221f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f 2023-04-07 01:53:53,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f, entries=150, sequenceid=203, filesize=11.8 K 2023-04-07 01:53:53,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/43e7f54812194461a484fa95decfe248 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248 2023-04-07 01:53:53,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248, entries=150, sequenceid=203, filesize=11.8 K 2023-04-07 01:53:53,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e2fe8c1647b59c84578d50aa3cf4039c in 961ms, sequenceid=203, compaction requested=true 2023-04-07 01:53:53,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:53,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:53,453 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:53,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:53,453 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:53,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:53,455 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41162 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:53,455 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:53,455 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:53,455 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:53,455 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,455 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,455 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/11559caf4382417e9c158cd0257265df, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=40.2 K 2023-04-07 01:53:53,456 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3e246d43ce9e4d428ea9a05be5f1fc06, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=35.5 K 2023-04-07 01:53:53,456 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 11559caf4382417e9c158cd0257265df, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:53,456 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3e246d43ce9e4d428ea9a05be5f1fc06, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:53,464 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4f63fb0e25db4fbe8bd31a6a802dc153, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680832431741 2023-04-07 01:53:53,465 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8a1883e22bef4ae783d76463b9f92ab0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680832431742 2023-04-07 01:53:53,465 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 73566c1c2f204e5dbb4776cd1b5031fd, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:53,466 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 55e4b92a8d574b97ab84583c4ce9221f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:53,475 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:53,488 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:53,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:53,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:53,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:53,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:53,495 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:53,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:53,500 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:53,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:53,512 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:53,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:53,562 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/a10cf79358dd492ca281eb03cb757c28 2023-04-07 01:53:53,566 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/1b59c4f0076448ce9149bf761b8a482f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1b59c4f0076448ce9149bf761b8a482f 2023-04-07 01:53:53,584 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into 1b59c4f0076448ce9149bf761b8a482f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:53,584 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:53,584 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=13, startTime=1680832433453; duration=0sec 2023-04-07 01:53:53,584 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:53,584 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:53,586 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:53,586 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:53,586 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:53,586 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/6c87601642794c8283960d1aedf4e793, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=35.5 K 2023-04-07 01:53:53,587 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6c87601642794c8283960d1aedf4e793, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680832431494 2023-04-07 01:53:53,587 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e68ac8c11eae47af89d756e10cfcafb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680832431742 2023-04-07 01:53:53,588 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 43e7f54812194461a484fa95decfe248, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:53,603 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/aac3cc258e8c4f35896660eb55fe5146 2023-04-07 01:53:53,605 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:53,627 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/bfecef37ce1e48dfbdc59cb4e24a61e3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/bfecef37ce1e48dfbdc59cb4e24a61e3 2023-04-07 01:53:53,636 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into bfecef37ce1e48dfbdc59cb4e24a61e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:53,636 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:53,636 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=13, startTime=1680832433453; duration=0sec 2023-04-07 01:53:53,637 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:53,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:53,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:53,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:53,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832493725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832493828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832493899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:53,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832493901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:53,931 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/b32613870a7c47a0b044dc1ba45e191d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/b32613870a7c47a0b044dc1ba45e191d 2023-04-07 01:53:53,940 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into b32613870a7c47a0b044dc1ba45e191d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:53,940 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:53,940 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=13, startTime=1680832433453; duration=0sec 2023-04-07 01:53:53,940 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:54,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832494022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38416 deadline: 1680832494031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,034 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:54,036 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/c09d3d4ae54d4cb79080f661f8ed5066 2023-04-07 01:53:54,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832494035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,041 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:54,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/a10cf79358dd492ca281eb03cb757c28 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28 2023-04-07 01:53:54,056 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28, entries=150, sequenceid=224, filesize=11.8 K 2023-04-07 01:53:54,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/aac3cc258e8c4f35896660eb55fe5146 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146 2023-04-07 01:53:54,075 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146, entries=150, sequenceid=224, filesize=11.8 K 2023-04-07 01:53:54,076 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/c09d3d4ae54d4cb79080f661f8ed5066 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066 2023-04-07 01:53:54,084 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066, entries=150, sequenceid=224, filesize=11.8 K 2023-04-07 01:53:54,085 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e2fe8c1647b59c84578d50aa3cf4039c in 573ms, sequenceid=224, compaction requested=false 2023-04-07 01:53:54,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:54,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:54,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:54,091 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:54,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aca0b78[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:54,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:54,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,095 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:54,095 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:54,097 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:54,106 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:54,107 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:54,107 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:54,107 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:54,107 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:54,109 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:54,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,112 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,114 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,117 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:54,117 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:54,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:54,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:54,117 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:54,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:54,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:54,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:54,117 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,118 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,118 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,118 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:54,117 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:54,118 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,118 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:54,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:54,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:54,234 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:54,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:54,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:54,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:54,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:54,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:54,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:54,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:54,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:54,251 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:54,251 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:54,252 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:53:54,252 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:54,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:54,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:54,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:54,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:54,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:54,315 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/9fba0d643b7f43fba230a54f42e35055 2023-04-07 01:53:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:54,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:54,386 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/6bb850e661c145c7a1a1e61b6c378a94 2023-04-07 01:53:54,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832494408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832494513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832494719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,844 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/2755775620c444deacb5619c6c2dd287 2023-04-07 01:53:54,867 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/9fba0d643b7f43fba230a54f42e35055 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055 2023-04-07 01:53:54,877 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055, entries=150, sequenceid=239, filesize=11.8 K 2023-04-07 01:53:54,879 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/6bb850e661c145c7a1a1e61b6c378a94 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94 2023-04-07 01:53:54,888 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94, entries=150, sequenceid=239, filesize=11.8 K 2023-04-07 01:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/2755775620c444deacb5619c6c2dd287 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287 2023-04-07 01:53:54,897 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287, entries=150, sequenceid=239, filesize=11.8 K 2023-04-07 01:53:54,899 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e2fe8c1647b59c84578d50aa3cf4039c in 647ms, sequenceid=239, compaction requested=true 2023-04-07 01:53:54,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:54,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:54,903 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e05a38[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:54,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:54,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:54,910 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,910 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:54,912 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:54,916 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:54,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:54,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:54,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:54,916 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:54,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:54,924 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:54,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:54,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:54,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:54,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:54,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:54,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:54,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:54,935 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:54,935 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:54,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:55,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:55,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:55,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:55,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:55,044 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,046 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:55,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:55,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/af8520c56332413fafbdce8dea69bdaa 2023-04-07 01:53:55,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e813739[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:55,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:55,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,060 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:55,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:55,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832495062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:55,066 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:55,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:55,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:55,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:55,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:55,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/cca0a44cd4524c2d9d1b58864b84a21d 2023-04-07 01:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:55,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:55,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,078 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:55,078 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:55,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832495165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,178 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:55,182 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:55,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:55,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:55,183 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,183 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,184 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:55,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:55,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:55,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:55,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:55,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:55,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bda8fc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:55,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:55,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:55,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:55,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:55,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:55,201 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:55,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:55,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:55,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:55,201 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:55,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,209 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:55,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,209 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:55,209 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:55,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:55,309 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:55,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:55,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:55,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:55,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:55,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:55,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:55,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:55,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:55,321 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,321 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:55,321 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:55,321 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:55,324 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e76ecd8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:55,327 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:55,327 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:55,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:55,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:55,336 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:55,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:55,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:55,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:55,337 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:55,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:55,348 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:55,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:55,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832495371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:55,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:55,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:55,454 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:55,454 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,454 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:55,454 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:55,455 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,455 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:55,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:55,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:55,459 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,459 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:55,459 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:55,459 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:55,461 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492052b4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:55,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:55,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,467 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:55,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:55,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:55,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:55,472 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:55,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:55,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:55,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:55,472 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:55,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:55,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:55,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:55,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:55,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,479 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:55,479 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:55,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:55,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:55,489 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/3306e0accc5c46d0a56cdb2e5f290b9e 2023-04-07 01:53:55,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/af8520c56332413fafbdce8dea69bdaa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa 2023-04-07 01:53:55,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa, entries=200, sequenceid=260, filesize=14.2 K 2023-04-07 01:53:55,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/cca0a44cd4524c2d9d1b58864b84a21d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d 2023-04-07 01:53:55,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d, entries=150, sequenceid=260, filesize=11.9 K 2023-04-07 01:53:55,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/3306e0accc5c46d0a56cdb2e5f290b9e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e 2023-04-07 01:53:55,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e, entries=150, sequenceid=260, filesize=11.9 K 2023-04-07 01:53:55,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for e2fe8c1647b59c84578d50aa3cf4039c in 502ms, sequenceid=260, compaction requested=true 2023-04-07 01:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:55,528 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:55,528 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:55,530 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:55,530 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:55,530 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,530 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1b59c4f0076448ce9149bf761b8a482f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=47.5 K 2023-04-07 01:53:55,531 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51060 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:55,531 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:55,531 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,531 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/b32613870a7c47a0b044dc1ba45e191d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=49.9 K 2023-04-07 01:53:55,531 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1b59c4f0076448ce9149bf761b8a482f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:55,533 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b32613870a7c47a0b044dc1ba45e191d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:55,535 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a10cf79358dd492ca281eb03cb757c28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680832432541 2023-04-07 01:53:55,535 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting aac3cc258e8c4f35896660eb55fe5146, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680832432541 2023-04-07 01:53:55,536 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9fba0d643b7f43fba230a54f42e35055, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680832433704 2023-04-07 01:53:55,537 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6bb850e661c145c7a1a1e61b6c378a94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680832433704 2023-04-07 01:53:55,537 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting af8520c56332413fafbdce8dea69bdaa, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434358 2023-04-07 01:53:55,537 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting cca0a44cd4524c2d9d1b58864b84a21d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434399 2023-04-07 01:53:55,549 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:55,552 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:55,564 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/16e3d1448f4c48858f867866aeaba105 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/16e3d1448f4c48858f867866aeaba105 2023-04-07 01:53:55,572 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/6e42ff8fb78c44ea85b993976c7f5791 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e42ff8fb78c44ea85b993976c7f5791 2023-04-07 01:53:55,574 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 16e3d1448f4c48858f867866aeaba105(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:55,574 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:55,574 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=12, startTime=1680832435527; duration=0sec 2023-04-07 01:53:55,574 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:55,574 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:53:55,576 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:53:55,576 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:55,577 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,577 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/bfecef37ce1e48dfbdc59cb4e24a61e3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=47.5 K 2023-04-07 01:53:55,577 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting bfecef37ce1e48dfbdc59cb4e24a61e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680832432332 2023-04-07 01:53:55,578 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c09d3d4ae54d4cb79080f661f8ed5066, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680832432541 2023-04-07 01:53:55,579 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into 6e42ff8fb78c44ea85b993976c7f5791(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:55,581 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:55,581 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2755775620c444deacb5619c6c2dd287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680832433704 2023-04-07 01:53:55,581 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:55,581 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=12, startTime=1680832435528; duration=0sec 2023-04-07 01:53:55,581 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:55,582 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3306e0accc5c46d0a56cdb2e5f290b9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434399 2023-04-07 01:53:55,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:55,586 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:55,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:55,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:55,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:55,590 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:55,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:55,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:55,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:55,592 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:55,592 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:55,604 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:53:55,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:55,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:55,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:55,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:55,605 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:55,605 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:55,608 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:55,625 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/24fbc314d70a4cf985036fdf5a18fcc1 2023-04-07 01:53:55,633 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/a1975325ac3b4388a73ece3a60e3a4ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a1975325ac3b4388a73ece3a60e3a4ec 2023-04-07 01:53:55,642 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into a1975325ac3b4388a73ece3a60e3a4ec(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:55,642 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:55,642 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=12, startTime=1680832435528; duration=0sec 2023-04-07 01:53:55,643 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:55,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:55,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832435737, max timestamp=1680832435750Num uniques 859; 2023-04-07 01:53:55,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:55,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832435737, max timestamp=1680832435750Num uniques 859; 2023-04-07 01:53:55,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:55,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832435737, max timestamp=1680832435750Num uniques 859; 2023-04-07 01:53:55,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:53:55,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832495803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832495915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832495922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,925 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:55,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:55,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832495928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:55,931 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:56,052 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/283c60faec1a4116b30cb61d21b2b255 2023-04-07 01:53:56,068 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/99c3e54f83c94885857aa156dfb2e0ff 2023-04-07 01:53:56,075 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/24fbc314d70a4cf985036fdf5a18fcc1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1 2023-04-07 01:53:56,095 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1, entries=150, sequenceid=273, filesize=11.9 K 2023-04-07 01:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/283c60faec1a4116b30cb61d21b2b255 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255 2023-04-07 01:53:56,106 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255, entries=150, sequenceid=273, filesize=11.9 K 2023-04-07 01:53:56,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/99c3e54f83c94885857aa156dfb2e0ff as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff 2023-04-07 01:53:56,117 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff, entries=150, sequenceid=273, filesize=11.9 K 2023-04-07 01:53:56,119 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e2fe8c1647b59c84578d50aa3cf4039c in 515ms, sequenceid=273, compaction requested=false 2023-04-07 01:53:56,119 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:56,120 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:56,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:56,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:56,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:56,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eaf8c56[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:56,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:56,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,132 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:56,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:56,132 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,132 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:56,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:56,133 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:56,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:56,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:56,138 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:56,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:56,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:56,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:56,138 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:56,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,155 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,155 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:56,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,156 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:56,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:56,156 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:56,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/79fa55421e9f49de8c64d2029b4e0871 2023-04-07 01:53:56,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832496167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,257 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:56,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:56,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:56,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:56,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:56,263 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:56,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:56,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:56,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:56,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:56,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:56,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:56,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:56,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:56,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:56,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674c3137[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:56,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:56,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832496271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:56,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:56,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:56,281 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:56,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:56,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:56,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:56,281 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:56,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,285 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:56,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:56,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:56,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:56,391 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:56,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:56,395 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:56,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:56,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:56,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:56,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:56,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:56,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:56,402 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,402 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:56,402 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:56,402 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:56,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:56,404 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e45012a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:56,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:56,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:56,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:56,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:56,416 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:56,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:56,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:56,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:56,416 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:56,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:56,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:56,423 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:56,423 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:56,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:56,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832496476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,523 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:56,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:56,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:56,527 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:56,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:56,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:56,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:56,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:56,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:56,533 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,533 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:56,533 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:56,533 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:56,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353b49a4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:56,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:56,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,542 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:56,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:56,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:56,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:56,547 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:56,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:56,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:56,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:56,547 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:56,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:56,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:56,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:56,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:56,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:56,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:56,558 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:56,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:56,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:56,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:56,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/08f747420db948578d7941cdd850f740 2023-04-07 01:53:56,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/5ffbb9f242b84b9881f88147b7cc5ad8 2023-04-07 01:53:56,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/79fa55421e9f49de8c64d2029b4e0871 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871 2023-04-07 01:53:56,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 01:53:56,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/08f747420db948578d7941cdd850f740 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740 2023-04-07 01:53:56,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 01:53:56,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/5ffbb9f242b84b9881f88147b7cc5ad8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8 2023-04-07 01:53:56,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:56,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:56,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 01:53:56,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:56,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e2fe8c1647b59c84578d50aa3cf4039c in 535ms, sequenceid=311, compaction requested=true 2023-04-07 01:53:56,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:56,664 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:56,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:56,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:56,664 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:56,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:56,664 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:56,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:56,664 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:56,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:56,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:56,665 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:56,665 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:56,665 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,666 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/16e3d1448f4c48858f867866aeaba105, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=36.2 K 2023-04-07 01:53:56,666 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:56,666 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:56,666 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,666 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e42ff8fb78c44ea85b993976c7f5791, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=36.2 K 2023-04-07 01:53:56,666 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:56,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:56,667 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6e42ff8fb78c44ea85b993976c7f5791, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434399 2023-04-07 01:53:56,666 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 16e3d1448f4c48858f867866aeaba105, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434399 2023-04-07 01:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,667 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 283c60faec1a4116b30cb61d21b2b255, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680832435043 2023-04-07 01:53:56,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:56,668 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 24fbc314d70a4cf985036fdf5a18fcc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680832435043 2023-04-07 01:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:56,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:56,669 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 08f747420db948578d7941cdd850f740, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:56,669 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 79fa55421e9f49de8c64d2029b4e0871, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:56,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:56,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:56,676 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,676 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:56,676 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:53:56,676 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:56,682 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:56,697 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#56 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:56,725 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/6b774e0d04ae488da7ee7fb3c568a03e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6b774e0d04ae488da7ee7fb3c568a03e 2023-04-07 01:53:56,728 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/47fc946b8d8d4396bbe1b04718bb0691 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/47fc946b8d8d4396bbe1b04718bb0691 2023-04-07 01:53:56,743 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 6b774e0d04ae488da7ee7fb3c568a03e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:56,743 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:56,743 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=13, startTime=1680832436664; duration=0sec 2023-04-07 01:53:56,743 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:56,743 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:56,745 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:56,745 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:56,746 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:56,746 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a1975325ac3b4388a73ece3a60e3a4ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=36.2 K 2023-04-07 01:53:56,746 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a1975325ac3b4388a73ece3a60e3a4ec, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680832434399 2023-04-07 01:53:56,748 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 99c3e54f83c94885857aa156dfb2e0ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680832435043 2023-04-07 01:53:56,749 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5ffbb9f242b84b9881f88147b7cc5ad8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:56,752 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into 47fc946b8d8d4396bbe1b04718bb0691(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:56,752 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:56,752 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=13, startTime=1680832436664; duration=0sec 2023-04-07 01:53:56,753 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:56,762 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:56,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:56,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832436807, max timestamp=1680832436829Num uniques 859; 2023-04-07 01:53:56,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832436807, max timestamp=1680832436829Num uniques 859; 2023-04-07 01:53:56,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:56,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832436807, max timestamp=1680832436829Num uniques 859; 2023-04-07 01:53:56,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:56,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:56,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832496878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:56,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:56,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832496983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,111 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/c0ec253ef0b44d158a00a8983688501e 2023-04-07 01:53:57,124 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8e557c3c238f48149c09dc939515d784 2023-04-07 01:53:57,136 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a 2023-04-07 01:53:57,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/c0ec253ef0b44d158a00a8983688501e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e 2023-04-07 01:53:57,147 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e, entries=100, sequenceid=321, filesize=9.5 K 2023-04-07 01:53:57,148 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/8e557c3c238f48149c09dc939515d784 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784 2023-04-07 01:53:57,153 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784, entries=100, sequenceid=321, filesize=9.5 K 2023-04-07 01:53:57,154 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a 2023-04-07 01:53:57,159 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a, entries=100, sequenceid=321, filesize=9.5 K 2023-04-07 01:53:57,160 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e2fe8c1647b59c84578d50aa3cf4039c in 484ms, sequenceid=321, compaction requested=false 2023-04-07 01:53:57,160 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,160 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f4ae69[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,165 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,166 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,166 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,167 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,170 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,170 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,174 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,181 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,181 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:57,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-07 01:53:57,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:57,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:57,196 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/4c900183036843fca5e88f290b65a075 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4c900183036843fca5e88f290b65a075 2023-04-07 01:53:57,207 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 4c900183036843fca5e88f290b65a075(size=12.5 K), total size for store is 22.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:57,207 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,207 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=13, startTime=1680832436665; duration=0sec 2023-04-07 01:53:57,207 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:57,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:57,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832497215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/3e5c296999e2448bb5a3b10bd2e843bf 2023-04-07 01:53:57,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/2ac7a2cd4469472eb479f6c0a971fe91 2023-04-07 01:53:57,281 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:57,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:57,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:57,296 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,298 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:57,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:57,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:57,308 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:57,315 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,318 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3137f429[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,325 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,328 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,328 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832497328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,343 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,343 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,343 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,343 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:57,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,443 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:57,449 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,451 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:57,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:57,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:57,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:57,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,455 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672dff74[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,477 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,477 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,483 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,483 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,483 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,491 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,491 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:57,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832497531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,591 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:57,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:57,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:57,596 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:57,596 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,596 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:57,596 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:57,596 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,596 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,598 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,598 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:57,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:57,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:57,599 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,600 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:57,600 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:57,600 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@834fc5c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,603 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,604 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,605 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,610 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,610 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,619 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,619 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,719 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:57,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:57,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:57,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:57,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:57,723 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:57,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:57,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:57,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,725 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:57,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:57,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:57,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:57,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,730 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,730 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,731 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa18cd5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/e5a7bcd21b1646d380f261b8a3e67a9e 2023-04-07 01:53:57,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,736 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,736 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,736 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,736 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/3e5c296999e2448bb5a3b10bd2e843bf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf 2023-04-07 01:53:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,743 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,744 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf, entries=200, sequenceid=360, filesize=14.3 K 2023-04-07 01:53:57,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/2ac7a2cd4469472eb479f6c0a971fe91 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91 2023-04-07 01:53:57,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,749 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91, entries=150, sequenceid=360, filesize=11.9 K 2023-04-07 01:53:57,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/e5a7bcd21b1646d380f261b8a3e67a9e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,756 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,759 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e, entries=150, sequenceid=360, filesize=11.9 K 2023-04-07 01:53:57,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e2fe8c1647b59c84578d50aa3cf4039c in 573ms, sequenceid=360, compaction requested=true 2023-04-07 01:53:57,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:57,762 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:57,762 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:57,763 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34728 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:57,763 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37168 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:57,763 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:53:57,763 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:53:57,763 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,763 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,764 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/47fc946b8d8d4396bbe1b04718bb0691, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=33.9 K 2023-04-07 01:53:57,764 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6b774e0d04ae488da7ee7fb3c568a03e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=36.3 K 2023-04-07 01:53:57,764 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 47fc946b8d8d4396bbe1b04718bb0691, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:57,764 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6b774e0d04ae488da7ee7fb3c568a03e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:57,765 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c0ec253ef0b44d158a00a8983688501e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832436159 2023-04-07 01:53:57,765 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8e557c3c238f48149c09dc939515d784, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832436159 2023-04-07 01:53:57,766 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3e5c296999e2448bb5a3b10bd2e843bf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:53:57,767 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2ac7a2cd4469472eb479f6c0a971fe91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:53:57,775 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:57,777 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:57,791 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/bd0efe8648344612a3e3451e8a0e5937 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/bd0efe8648344612a3e3451e8a0e5937 2023-04-07 01:53:57,796 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/77374d28aa1b4d9ea0bd8bf5893b762f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/77374d28aa1b4d9ea0bd8bf5893b762f 2023-04-07 01:53:57,798 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into bd0efe8648344612a3e3451e8a0e5937(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:57,798 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,798 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=13, startTime=1680832437762; duration=0sec 2023-04-07 01:53:57,798 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:53:57,798 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:53:57,801 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34728 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:53:57,802 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:53:57,802 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,802 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4c900183036843fca5e88f290b65a075, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=33.9 K 2023-04-07 01:53:57,803 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4c900183036843fca5e88f290b65a075, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680832435789 2023-04-07 01:53:57,804 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2e2fbd858c9e4b3aa7d3bd2382e1a24a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832436159 2023-04-07 01:53:57,804 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e5a7bcd21b1646d380f261b8a3e67a9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:53:57,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 77374d28aa1b4d9ea0bd8bf5893b762f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:57,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=13, startTime=1680832437762; duration=0sec 2023-04-07 01:53:57,808 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:57,817 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#65 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:53:57,828 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/913f7c64c5bd46eebdf4ff248c7a8ea3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/913f7c64c5bd46eebdf4ff248c7a8ea3 2023-04-07 01:53:57,835 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 913f7c64c5bd46eebdf4ff248c7a8ea3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:53:57,835 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:57,835 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=13, startTime=1680832437762; duration=0sec 2023-04-07 01:53:57,835 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:53:57,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:57,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:53:57,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:57,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:57,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:57,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:57,864 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:57,864 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,865 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:57,865 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:57,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,869 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,869 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,869 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,869 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:57,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:57,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:57,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:57,893 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:57,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:57,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:57,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:57,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832437856, max timestamp=1680832437904Num uniques 859; 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832437856, max timestamp=1680832437904Num uniques 859; 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832437856, max timestamp=1680832437904Num uniques 859; 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:57,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:57,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:53:57,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:57,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1439b514[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:57,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:57,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:57,911 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:57,917 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:57,918 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:57,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,929 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:57,929 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:57,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:57,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:57,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:57,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:57,930 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:57,930 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:57,929 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:57,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:57,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832497970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,034 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,035 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,035 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,036 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,039 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,039 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,039 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,039 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,040 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c672eb8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:58,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38354 deadline: 1680832498043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,047 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:58,047 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,049 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:58,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,052 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,053 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:58,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38430 deadline: 1680832498050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,055 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:53:58,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,059 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,065 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,065 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:58,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832498073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,165 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,169 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339ee774[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,193 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,193 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,198 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,201 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832498277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/aaf244424ca745db913cd4b3f4e95d31 2023-04-07 01:53:58,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,308 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,308 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,308 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,308 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,308 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,308 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cf02f6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,333 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,335 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,335 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,338 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,342 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,343 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,443 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,447 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,449 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c07924e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,456 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,456 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,457 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,458 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,458 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,459 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,462 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,463 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,464 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,464 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,464 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,575 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832498585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,599 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59307198[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,602 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,602 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,605 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,613 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,613 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,630 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/b9ed66c7b74048439d3238411c3aeace 2023-04-07 01:53:58,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,734 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,734 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,734 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,734 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,736 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,736 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,745 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,745 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:58,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:58,759 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/0705225b15e045d2a5a59358e30fdb29 2023-04-07 01:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d51888d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:58,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:58,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/aaf244424ca745db913cd4b3f4e95d31 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31 2023-04-07 01:53:58,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:58,780 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,780 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:58,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31, entries=200, sequenceid=374, filesize=14.3 K 2023-04-07 01:53:58,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/b9ed66c7b74048439d3238411c3aeace as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace 2023-04-07 01:53:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:58,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:58,800 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:58,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:58,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:58,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:58,800 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:58,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:58,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace, entries=150, sequenceid=374, filesize=11.9 K 2023-04-07 01:53:58,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/0705225b15e045d2a5a59358e30fdb29 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29 2023-04-07 01:53:58,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:58,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:58,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:58,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:58,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:58,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:58,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:58,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:58,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29, entries=150, sequenceid=374, filesize=11.9 K 2023-04-07 01:53:58,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:58,819 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:58,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:58,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e2fe8c1647b59c84578d50aa3cf4039c in 984ms, sequenceid=374, compaction requested=false 2023-04-07 01:53:58,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:58,920 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:58,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:58,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:58,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:58,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:58,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:58,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,925 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:58,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:58,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:58,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:58,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:58,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:58,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:58,927 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:58,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:58,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:58,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:53:59,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:59,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:59,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832499113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832499223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,338 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/1f368c858eca489da55de56ba439c848 2023-04-07 01:53:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832499428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:53:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832499734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,751 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/6e4060d78aa74c6993a276da793d4ffe 2023-04-07 01:53:59,769 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/79517683c2974d6abf0377e4b5a7456f 2023-04-07 01:53:59,778 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/1f368c858eca489da55de56ba439c848 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848 2023-04-07 01:53:59,784 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848, entries=150, sequenceid=410, filesize=11.9 K 2023-04-07 01:53:59,785 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/6e4060d78aa74c6993a276da793d4ffe as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe 2023-04-07 01:53:59,791 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe, entries=150, sequenceid=410, filesize=11.9 K 2023-04-07 01:53:59,792 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/79517683c2974d6abf0377e4b5a7456f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f 2023-04-07 01:53:59,806 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f, entries=150, sequenceid=410, filesize=11.9 K 2023-04-07 01:53:59,807 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e2fe8c1647b59c84578d50aa3cf4039c in 880ms, sequenceid=410, compaction requested=true 2023-04-07 01:53:59,807 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:53:59,807 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:59,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:59,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:59,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:59,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:59,812 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4054e163[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:59,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:59,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,823 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:59,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:59,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:59,829 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:59,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:59,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:59,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:59,829 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:59,830 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:59,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:59,831 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,833 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:59,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:53:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:59,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:59,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:59,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:59,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:53:59,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:59,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:53:59,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:53:59,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:53:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:53:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:53:59,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:53:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:53:59,940 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:53:59,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:59,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:53:59,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:53:59,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:53:59,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:59,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:53:59,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:59,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:53:59,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:53:59,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:59,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:59,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:53:59,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:53:59,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:53:59,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:53:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:53:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:53:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:53:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:53:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:53:59,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:59,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:53:59,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:53:59,955 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:53:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:53:59,966 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ec5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:53:59,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:53:59,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,971 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:53:59,972 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,972 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:53:59,973 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:53:59,978 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:53:59,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:53:59,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:53:59,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:53:59,978 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:53:59,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:53:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:53:59,983 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:53:59,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:53:59,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:53:59,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:53:59,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:53:59,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/d913351408e54d0d9ab8e782350f2866 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:53:59,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:53:59,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:53:59,988 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:53:59,988 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832439995, max timestamp=1680832440021Num uniques 859; 2023-04-07 01:54:00,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:00,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832439995, max timestamp=1680832440021Num uniques 859; 2023-04-07 01:54:00,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832439995, max timestamp=1680832440021Num uniques 859; 2023-04-07 01:54:00,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:00,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:00,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ce0717792d264b4fa4a9180a80662d3d 2023-04-07 01:54:00,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,100 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dff5ba3d49b34a7087f3a1067c8b7ab2 2023-04-07 01:54:00,103 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,104 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/d913351408e54d0d9ab8e782350f2866 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866 2023-04-07 01:54:00,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866, entries=200, sequenceid=421, filesize=14.3 K 2023-04-07 01:54:00,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ce0717792d264b4fa4a9180a80662d3d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d 2023-04-07 01:54:00,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832500131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,144 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d, entries=150, sequenceid=421, filesize=11.9 K 2023-04-07 01:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dff5ba3d49b34a7087f3a1067c8b7ab2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2 2023-04-07 01:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:00,149 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae783ec[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:00,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:00,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:00,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:00,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:00,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:00,164 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:00,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:00,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:00,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:00,164 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:00,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2, entries=150, sequenceid=421, filesize=11.9 K 2023-04-07 01:54:00,165 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,166 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,166 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e2fe8c1647b59c84578d50aa3cf4039c in 226ms, sequenceid=421, compaction requested=true 2023-04-07 01:54:00,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:00,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:00,167 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:00,167 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:00,166 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,169 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54340 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,169 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:00,169 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:54:00,169 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,170 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/bd0efe8648344612a3e3451e8a0e5937, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=48.3 K 2023-04-07 01:54:00,169 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,170 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,170 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/77374d28aa1b4d9ea0bd8bf5893b762f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=53.1 K 2023-04-07 01:54:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,170 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bd0efe8648344612a3e3451e8a0e5937, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:54:00,171 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b9ed66c7b74048439d3238411c3aeace, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680832437201 2023-04-07 01:54:00,171 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 77374d28aa1b4d9ea0bd8bf5893b762f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:54:00,171 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6e4060d78aa74c6993a276da793d4ffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680832437931 2023-04-07 01:54:00,171 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:00,172 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting aaf244424ca745db913cd4b3f4e95d31, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680832437201 2023-04-07 01:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:00,172 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ce0717792d264b4fa4a9180a80662d3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439106 2023-04-07 01:54:00,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:00,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:00,173 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1f368c858eca489da55de56ba439c848, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680832437931 2023-04-07 01:54:00,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,173 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,173 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,173 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d913351408e54d0d9ab8e782350f2866, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439103 2023-04-07 01:54:00,188 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:00,192 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:00,227 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/5c62cc1963ff46f097c7a9530efe0262 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/5c62cc1963ff46f097c7a9530efe0262 2023-04-07 01:54:00,234 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 5c62cc1963ff46f097c7a9530efe0262(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:00,234 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:00,234 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=12, startTime=1680832440166; duration=0sec 2023-04-07 01:54:00,234 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:00,234 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:00,236 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:00,236 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:54:00,237 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,237 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/913f7c64c5bd46eebdf4ff248c7a8ea3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=48.3 K 2023-04-07 01:54:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:00,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-07 01:54:00,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:00,241 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 913f7c64c5bd46eebdf4ff248c7a8ea3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680832436855 2023-04-07 01:54:00,242 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0705225b15e045d2a5a59358e30fdb29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680832437201 2023-04-07 01:54:00,243 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 79517683c2974d6abf0377e4b5a7456f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680832437931 2023-04-07 01:54:00,243 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting dff5ba3d49b34a7087f3a1067c8b7ab2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439106 2023-04-07 01:54:00,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832500251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832500256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,267 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:00,272 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/da3d5e27bb304673acf5fba8b27ae727 2023-04-07 01:54:00,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,296 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,298 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,298 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:00,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:00,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b637ddd[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:00,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:00,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:00,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:00,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,314 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/917b39a67837449693d8d998c0824548 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/917b39a67837449693d8d998c0824548 2023-04-07 01:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:00,317 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:00,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:00,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:00,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:00,317 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:00,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 917b39a67837449693d8d998c0824548(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:00,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:00,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=12, startTime=1680832440167; duration=0sec 2023-04-07 01:54:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:00,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:00,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:00,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,330 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,330 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832500359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832500363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,430 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,440 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,440 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,440 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,440 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:00,453 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:00,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5bc22b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:00,455 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:00,455 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,457 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:00,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:00,459 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:00,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:00,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:00,460 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:00,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:00,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:00,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:00,460 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:00,463 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,463 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,463 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:00,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:00,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:00,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:00,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38422 deadline: 1680832500564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832500565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,574 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,576 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,579 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,579 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,579 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,580 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:00,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202006c3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:00,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:00,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,586 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:00,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:00,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:00,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:00,591 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:00,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:00,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:00,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:00,591 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:00,593 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,593 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,595 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:00,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:00,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:00,602 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,623 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/d4c3a6eaa174407589134df4ce31fecd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d4c3a6eaa174407589134df4ce31fecd 2023-04-07 01:54:00,629 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into d4c3a6eaa174407589134df4ce31fecd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:00,629 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:00,629 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=12, startTime=1680832440167; duration=0sec 2023-04-07 01:54:00,629 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:00,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/a438adc5f81a405abfef53b11e2682ee 2023-04-07 01:54:00,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,720 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/41b9b34af5884616aca405f9487fadb7 2023-04-07 01:54:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,726 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,726 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,726 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,726 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:00,728 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/da3d5e27bb304673acf5fba8b27ae727 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727 2023-04-07 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46be09c8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:00,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:00,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:00,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:00,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:00,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:00,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:00,737 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727, entries=200, sequenceid=458, filesize=14.3 K 2023-04-07 01:54:00,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/a438adc5f81a405abfef53b11e2682ee as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee 2023-04-07 01:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:00,743 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:00,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:00,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:00,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:00,743 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:00,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:00,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:00,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,751 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:00,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:00,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:00,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee, entries=150, sequenceid=458, filesize=11.9 K 2023-04-07 01:54:00,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:00,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:00,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:00,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/41b9b34af5884616aca405f9487fadb7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7 2023-04-07 01:54:00,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7, entries=150, sequenceid=458, filesize=11.9 K 2023-04-07 01:54:00,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e2fe8c1647b59c84578d50aa3cf4039c in 529ms, sequenceid=458, compaction requested=false 2023-04-07 01:54:00,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:00,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:00,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:00,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:00,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:00,856 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:00,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:00,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:00,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,858 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:00,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:00,859 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,859 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:00,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:00,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:00,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:00,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:00,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:00,861 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:00,861 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:00,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832440876, max timestamp=1680832440888Num uniques 859; 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832440876, max timestamp=1680832440888Num uniques 859; 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832440876, max timestamp=1680832440888Num uniques 859; 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:00,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832500908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832500910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832500910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832501011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832501015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38436 deadline: 1680832501015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832501214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832501218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832501219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,271 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/66e8eff59b2242dd956e92ccae777bf2 2023-04-07 01:54:01,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832501516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832501521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:01,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832501522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:01,692 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/7464ee1193cc4c999ed790c46177148f 2023-04-07 01:54:02,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:02,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832502020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:02,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832502025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:02,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832502028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,109 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/30c5892bc09c4c16a2ee3e92ff0dd209 2023-04-07 01:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/66e8eff59b2242dd956e92ccae777bf2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2 2023-04-07 01:54:02,122 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2, entries=150, sequenceid=470, filesize=11.9 K 2023-04-07 01:54:02,123 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/7464ee1193cc4c999ed790c46177148f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f 2023-04-07 01:54:02,130 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f, entries=150, sequenceid=470, filesize=11.9 K 2023-04-07 01:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 01:54:02,131 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/30c5892bc09c4c16a2ee3e92ff0dd209 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209 2023-04-07 01:54:02,139 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209, entries=150, sequenceid=470, filesize=11.9 K 2023-04-07 01:54:02,140 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for e2fe8c1647b59c84578d50aa3cf4039c in 1279ms, sequenceid=470, compaction requested=true 2023-04-07 01:54:02,140 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:02,140 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:02,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:02,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e30957[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:02,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:02,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:02,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:02,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:02,151 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:02,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:02,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:02,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:02,152 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:02,155 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,155 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,155 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,157 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,160 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:02,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:02,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:02,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:02,161 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:02,161 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:02,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:02,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:02,261 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:02,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:02,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:02,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:02,264 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:02,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:02,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:02,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:02,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:02,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:02,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:02,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:02,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:02,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:02,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:02,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:02,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:02,268 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-07 01:54:02,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:02,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:02,268 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:02,269 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:02,269 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:02,269 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:02,285 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/c2485608d1824bb2b4afe384a0f9a3ec 2023-04-07 01:54:02,706 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ed6a4d341ebd4d1a9e2f71fbe13c2139 2023-04-07 01:54:02,754 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/b26b7b47f0f14d9c8d614bf7823da49d 2023-04-07 01:54:02,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/c2485608d1824bb2b4afe384a0f9a3ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec 2023-04-07 01:54:02,769 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:02,770 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ed6a4d341ebd4d1a9e2f71fbe13c2139 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139 2023-04-07 01:54:02,779 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:02,780 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/b26b7b47f0f14d9c8d614bf7823da49d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d 2023-04-07 01:54:02,793 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:02,794 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=0 B/0 for e2fe8c1647b59c84578d50aa3cf4039c in 526ms, sequenceid=506, compaction requested=true 2023-04-07 01:54:02,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:02,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:02,796 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:02,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dbdb3a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:02,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,800 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:02,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:02,806 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:02,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:02,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:02,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:02,806 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:02,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,817 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,817 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:02,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:02,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:02,817 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:02,817 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,817 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:02,817 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:02,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:02,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:02,917 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:02,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:02,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:02,925 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:02,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:02,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:02,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:02,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,928 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:02,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:02,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:02,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:02,932 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:02,932 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:02,932 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:02,932 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:02,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:02,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446ddf7c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:02,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:02,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:02,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:02,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:02,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:02,949 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:02,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:02,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:02,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:02,949 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:02,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:02,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:02,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:02,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:02,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:02,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:02,960 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:02,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:03,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:03,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:03,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:03,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:03,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:03,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:03,084 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:03,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:03,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:03,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:03,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,086 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:03,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:03,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:03,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832503148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832443134, max timestamp=1680832443154Num uniques 859; 2023-04-07 01:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832443134, max timestamp=1680832443154Num uniques 859; 2023-04-07 01:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:03,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:03,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2141a8af04ea4e37aa814d96c745e650 2023-04-07 01:54:03,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:03,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832443134, max timestamp=1680832443154Num uniques 859; 2023-04-07 01:54:03,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:03,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:03,160 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:03,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:03,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ab7e70[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:03,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:03,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,169 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:03,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:03,171 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:03,171 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:03,171 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:03,171 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:03,175 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:03,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:03,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:03,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:03,175 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:03,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:03,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,179 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:03,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:03,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:03,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:03,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:03,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:03,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:03,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:03,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,186 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:03,186 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:03,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:03,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:03,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:03,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:03,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/e4b2d0538e96402e9ddf5b8c6d8045a1 2023-04-07 01:54:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832503260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832503267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/063c620eb2b245e0a1a14eddb2421b81 2023-04-07 01:54:03,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832503269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/2141a8af04ea4e37aa814d96c745e650 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650 2023-04-07 01:54:03,286 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:03,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:03,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:03,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650, entries=150, sequenceid=518, filesize=11.9 K 2023-04-07 01:54:03,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/e4b2d0538e96402e9ddf5b8c6d8045a1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1 2023-04-07 01:54:03,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:03,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:03,303 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:03,303 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:03,303 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:03,303 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:03,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1, entries=150, sequenceid=518, filesize=11.9 K 2023-04-07 01:54:03,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:03,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/063c620eb2b245e0a1a14eddb2421b81 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81 2023-04-07 01:54:03,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:03,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:03,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:03,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81, entries=150, sequenceid=518, filesize=11.9 K 2023-04-07 01:54:03,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e2fe8c1647b59c84578d50aa3cf4039c in 275ms, sequenceid=518, compaction requested=true 2023-04-07 01:54:03,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:03,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:03,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:03,316 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:03,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:03,316 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:03,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:03,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:03,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:03,320 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:03,320 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:54:03,320 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,320 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d4c3a6eaa174407589134df4ce31fecd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=60.3 K 2023-04-07 01:54:03,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64222 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:03,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:54:03,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/5c62cc1963ff46f097c7a9530efe0262, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=62.7 K 2023-04-07 01:54:03,321 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d4c3a6eaa174407589134df4ce31fecd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439106 2023-04-07 01:54:03,322 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5c62cc1963ff46f097c7a9530efe0262, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439106 2023-04-07 01:54:03,322 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a438adc5f81a405abfef53b11e2682ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680832440127 2023-04-07 01:54:03,322 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting da3d5e27bb304673acf5fba8b27ae727, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680832440115 2023-04-07 01:54:03,322 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7464ee1193cc4c999ed790c46177148f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680832440242 2023-04-07 01:54:03,323 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 66e8eff59b2242dd956e92ccae777bf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680832440242 2023-04-07 01:54:03,323 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ed6a4d341ebd4d1a9e2f71fbe13c2139, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832440904 2023-04-07 01:54:03,323 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c2485608d1824bb2b4afe384a0f9a3ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832440904 2023-04-07 01:54:03,323 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e4b2d0538e96402e9ddf5b8c6d8045a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:03,324 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2141a8af04ea4e37aa814d96c745e650, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:03,328 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:03,328 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:03,328 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,328 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:03,328 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-07 01:54:03,328 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:03,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:03,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:03,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:03,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:03,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:03,351 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#90 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:03,375 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#92 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:03,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:03,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832503409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832503410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,442 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/c8913a3f1f6245d6b605e58fe657876c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c8913a3f1f6245d6b605e58fe657876c 2023-04-07 01:54:03,453 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into c8913a3f1f6245d6b605e58fe657876c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:03,453 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:03,453 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=11, startTime=1680832443316; duration=0sec 2023-04-07 01:54:03,453 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:03,453 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:03,457 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:03,457 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:54:03,458 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:03,458 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/917b39a67837449693d8d998c0824548, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=60.3 K 2023-04-07 01:54:03,458 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/e3c07426255444daa72399b9a5fde221 2023-04-07 01:54:03,459 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 917b39a67837449693d8d998c0824548, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680832439106 2023-04-07 01:54:03,460 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 41b9b34af5884616aca405f9487fadb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680832440127 2023-04-07 01:54:03,461 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/f8af63d20e084cefa45fe1357e57478b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/f8af63d20e084cefa45fe1357e57478b 2023-04-07 01:54:03,461 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 30c5892bc09c4c16a2ee3e92ff0dd209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680832440242 2023-04-07 01:54:03,462 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b26b7b47f0f14d9c8d614bf7823da49d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832440904 2023-04-07 01:54:03,464 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 063c620eb2b245e0a1a14eddb2421b81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:03,469 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into f8af63d20e084cefa45fe1357e57478b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:03,469 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:03,469 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=11, startTime=1680832443316; duration=0sec 2023-04-07 01:54:03,469 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832503474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,487 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#93 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:03,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832503516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832503516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,573 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/40b9778b6c024c56a0919588a4345e10 2023-04-07 01:54:03,579 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/8c5fe4daa0e94e53bc75f6e98d960e2d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/8c5fe4daa0e94e53bc75f6e98d960e2d 2023-04-07 01:54:03,588 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 8c5fe4daa0e94e53bc75f6e98d960e2d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:03,588 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:03,588 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=11, startTime=1680832443316; duration=0sec 2023-04-07 01:54:03,588 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:03,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832503720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832503720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:03,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:03,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832503789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,016 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dd8cc2b9f7d44b8ab46f62b23c063947 2023-04-07 01:54:04,022 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/e3c07426255444daa72399b9a5fde221 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221 2023-04-07 01:54:04,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832504023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832504024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,027 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221, entries=150, sequenceid=553, filesize=11.9 K 2023-04-07 01:54:04,028 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/40b9778b6c024c56a0919588a4345e10 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10 2023-04-07 01:54:04,032 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10, entries=150, sequenceid=553, filesize=11.9 K 2023-04-07 01:54:04,033 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dd8cc2b9f7d44b8ab46f62b23c063947 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947 2023-04-07 01:54:04,047 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947, entries=150, sequenceid=553, filesize=11.9 K 2023-04-07 01:54:04,048 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e2fe8c1647b59c84578d50aa3cf4039c in 720ms, sequenceid=553, compaction requested=false 2023-04-07 01:54:04,049 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:04,049 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:04,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0c78c3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:04,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:04,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:04,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:04,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:04,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:04,059 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:04,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:04,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:04,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:04,059 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:04,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,068 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:04,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:04,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,072 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:04,072 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:04,172 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:04,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:04,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:04,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:04,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:04,176 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:04,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:04,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:04,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:04,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:04,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:04,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:04,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:04,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:04,183 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:04,183 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:04,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:04,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832444330, max timestamp=1680832444350Num uniques 859; 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832444330, max timestamp=1680832444350Num uniques 859; 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832444330, max timestamp=1680832444350Num uniques 859; 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:04,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:04,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832504400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832504503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832504527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832504529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,599 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/9d63f9e9cc49435fb3d053a09d0596df 2023-04-07 01:54:04,627 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d 2023-04-07 01:54:04,670 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/7ce244d898104482a20281f12751312e 2023-04-07 01:54:04,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/9d63f9e9cc49435fb3d053a09d0596df as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df 2023-04-07 01:54:04,692 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df, entries=150, sequenceid=567, filesize=11.9 K 2023-04-07 01:54:04,693 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d 2023-04-07 01:54:04,698 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d, entries=150, sequenceid=567, filesize=11.9 K 2023-04-07 01:54:04,699 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/7ce244d898104482a20281f12751312e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e 2023-04-07 01:54:04,704 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e, entries=150, sequenceid=567, filesize=11.9 K 2023-04-07 01:54:04,705 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e2fe8c1647b59c84578d50aa3cf4039c in 522ms, sequenceid=567, compaction requested=true 2023-04-07 01:54:04,705 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:04,705 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:04,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:04,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b46e73[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:04,712 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:04,712 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:04,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:04,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:04,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,716 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:04,716 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:04,720 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:04,720 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,720 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:04,725 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:04,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:04,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:04,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:04,725 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:04,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,728 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:04,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:04,734 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:04,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:04,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:04,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/7e741a1a1e774721abeaba45f7d192e8 2023-04-07 01:54:04,735 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:04,735 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832504744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/1f073f949ac94067abcfaa71086d5679 2023-04-07 01:54:04,836 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:04,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:04,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:04,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:04,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:04,839 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:04,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,841 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:04,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:04,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:04,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:04,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:04,843 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:04,843 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:04,843 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:04,843 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:04,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:04,845 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:04,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c88cfe6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:04,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:04,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,848 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:04,848 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,848 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:04,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:04,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832504850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,851 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:04,853 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:04,853 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:04,853 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:04,853 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:04,854 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:04,855 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,855 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:04,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,855 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:04,855 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:04,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:04,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:04,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:04,859 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:04,859 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:04,859 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:04,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:04,860 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,860 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:04,860 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:04,963 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:04,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:04,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:04,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:04,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:04,975 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:04,975 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:04,975 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:04,975 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:04,975 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:04,975 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,022 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,023 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:05,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:05,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:05,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:05,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:05,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:05,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:05,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:05,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832505053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:05,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457974f9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:05,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:05,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dacacc40dc614c5bbc66b7644391bbf4 2023-04-07 01:54:05,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/7e741a1a1e774721abeaba45f7d192e8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8 2023-04-07 01:54:05,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8, entries=200, sequenceid=604, filesize=14.3 K 2023-04-07 01:54:05,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/1f073f949ac94067abcfaa71086d5679 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679 2023-04-07 01:54:05,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679, entries=150, sequenceid=604, filesize=11.9 K 2023-04-07 01:54:05,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dacacc40dc614c5bbc66b7644391bbf4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4 2023-04-07 01:54:05,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:05,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:05,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4, entries=150, sequenceid=604, filesize=11.9 K 2023-04-07 01:54:05,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e2fe8c1647b59c84578d50aa3cf4039c in 485ms, sequenceid=604, compaction requested=true 2023-04-07 01:54:05,200 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:05,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:05,200 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:05,200 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:05,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:05,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:05,202 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52206 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:05,202 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/A is initiating minor compaction (all files) 2023-04-07 01:54:05,202 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/A in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,202 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c8913a3f1f6245d6b605e58fe657876c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=51.0 K 2023-04-07 01:54:05,202 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:05,203 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/B is initiating minor compaction (all files) 2023-04-07 01:54:05,203 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/B in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,203 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/f8af63d20e084cefa45fe1357e57478b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=48.6 K 2023-04-07 01:54:05,203 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c8913a3f1f6245d6b605e58fe657876c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:05,203 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f8af63d20e084cefa45fe1357e57478b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:05,204 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e3c07426255444daa72399b9a5fde221, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832443233 2023-04-07 01:54:05,204 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 40b9778b6c024c56a0919588a4345e10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832443233 2023-04-07 01:54:05,204 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9d63f9e9cc49435fb3d053a09d0596df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680832443393 2023-04-07 01:54:05,205 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4f3e6a07a2ed41ffa5ab8e161ac4f53d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680832443393 2023-04-07 01:54:05,206 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7e741a1a1e774721abeaba45f7d192e8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680832444388 2023-04-07 01:54:05,206 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1f073f949ac94067abcfaa71086d5679, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680832444394 2023-04-07 01:54:05,225 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#A#compaction#102 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:05,225 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#B#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:05,252 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:05,252 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,252 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,252 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:05,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:05,256 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:05,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:05,257 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:05,257 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:05,257 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:05,260 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/a4b85046360f4ca1b0b4ef17c8a0a15f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a4b85046360f4ca1b0b4ef17c8a0a15f 2023-04-07 01:54:05,261 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/9b9e47d9a74243b9938feb5d70c90674 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9b9e47d9a74243b9938feb5d70c90674 2023-04-07 01:54:05,270 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/B of e2fe8c1647b59c84578d50aa3cf4039c into a4b85046360f4ca1b0b4ef17c8a0a15f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:05,270 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/A of e2fe8c1647b59c84578d50aa3cf4039c into 9b9e47d9a74243b9938feb5d70c90674(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:05,270 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:05,270 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:05,270 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/B, priority=12, startTime=1680832445200; duration=0sec 2023-04-07 01:54:05,271 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/A, priority=12, startTime=1680832445200; duration=0sec 2023-04-07 01:54:05,271 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:05,271 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:05,271 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:05,273 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:05,273 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): e2fe8c1647b59c84578d50aa3cf4039c/C is initiating minor compaction (all files) 2023-04-07 01:54:05,273 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2fe8c1647b59c84578d50aa3cf4039c/C in TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,273 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/8c5fe4daa0e94e53bc75f6e98d960e2d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp, totalSize=48.6 K 2023-04-07 01:54:05,274 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8c5fe4daa0e94e53bc75f6e98d960e2d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832443029 2023-04-07 01:54:05,274 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dd8cc2b9f7d44b8ab46f62b23c063947, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832443233 2023-04-07 01:54:05,275 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7ce244d898104482a20281f12751312e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680832443393 2023-04-07 01:54:05,276 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dacacc40dc614c5bbc66b7644391bbf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680832444394 2023-04-07 01:54:05,302 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): e2fe8c1647b59c84578d50aa3cf4039c#C#compaction#104 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:05,306 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,306 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,306 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,345 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/5031f38ec858405d8f3a0a505a3acd75 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5031f38ec858405d8f3a0a505a3acd75 2023-04-07 01:54:05,352 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2fe8c1647b59c84578d50aa3cf4039c/C of e2fe8c1647b59c84578d50aa3cf4039c into 5031f38ec858405d8f3a0a505a3acd75(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:05,352 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:05,352 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c., storeName=e2fe8c1647b59c84578d50aa3cf4039c/C, priority=12, startTime=1680832445200; duration=0sec 2023-04-07 01:54:05,352 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:05,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:05,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:05,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:05,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:05,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832445394, max timestamp=1680832445422Num uniques 859; 2023-04-07 01:54:05,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:05,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832445394, max timestamp=1680832445422Num uniques 859; 2023-04-07 01:54:05,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:05,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832445394, max timestamp=1680832445422Num uniques 859; 2023-04-07 01:54:05,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:05,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832505459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:05,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:05,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:05,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:05,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:05,502 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:05,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:05,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:05,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,505 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,505 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:05,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:05,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:05,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:05,506 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,506 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:05,506 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:05,506 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:05,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,507 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:05,507 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:05,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,509 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,509 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:05,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2cd6f5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:05,510 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:05,510 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,523 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:05,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:05,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:05,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:05,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:05,531 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:05,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:05,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:05,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:05,531 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38422 deadline: 1680832505534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,537 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38416 deadline: 1680832505539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:05,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:05,545 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:05,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:05,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832505560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:05,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:05,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:05,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:05,648 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:05,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:05,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:05,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:05,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:05,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:05,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:05,652 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,652 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:05,652 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:05,653 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:05,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:05,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:05,653 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:05,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:05,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f0efc35[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:05,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:05,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:05,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:05,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:05,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:05,663 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:05,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:05,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:05,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:05,663 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:05,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:05,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,670 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:05,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:05,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:05,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832505764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,771 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:05,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:05,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:05,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:05,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:05,774 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:05,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:05,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:05,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/fea009ac6ea84d479f30dbf7e35cd865 2023-04-07 01:54:05,775 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,775 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,777 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:05,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:05,785 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:05,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:05,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:05,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:05,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:05,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1553f3cd[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:05,790 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:05,790 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:05,793 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:05,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/a3cade0637e74c27928c3a9a762530b3 2023-04-07 01:54:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:05,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:05,797 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:05,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:05,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:05,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:05,797 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:05,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:05,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:05,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,811 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,811 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,811 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:05,811 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:05,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:05,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,911 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:05,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:05,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:05,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:05,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:05,914 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:05,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:05,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:05,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,916 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,916 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:05,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:05,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:05,918 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,918 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:05,918 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:05,918 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:05,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:05,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,920 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:05,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b038322[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:05,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:05,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,924 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:05,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:05,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:05,924 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:05,928 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,928 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,928 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,928 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:05,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:05,930 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:05,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:05,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:05,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:05,930 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:05,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:05,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:05,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:05,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:05,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:05,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:05,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:05,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:05,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:05,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:05,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:05,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,936 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:05,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:05,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:05,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:05,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:05,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:05,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:06,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:06,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:06,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:06,042 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:06,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:06,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:06,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,045 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:06,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:06,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:06,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:06,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:06,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:06,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:06,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:06,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:06,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:06,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272465c9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:06,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:06,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:06,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:06,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:06,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:06,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:06,057 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:06,060 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:06,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:06,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:06,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:06,060 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:06,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,068 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:06,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832506067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:06,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:06,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:06,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:06,073 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:06,073 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:06,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:06,173 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:06,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:06,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:06,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:06,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:06,195 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:06,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:06,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:06,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,197 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:06,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:06,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:06,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:06,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:06,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:06,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:06,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:06,201 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:06,202 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:06,202 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:06,202 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:06,204 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:06,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f83ca4c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:06,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:06,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:06,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:06,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:06,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:06,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:06,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:06,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/f6edaabb52b748e99e3d98abedb10e6e 2023-04-07 01:54:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:06,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:06,216 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:06,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:06,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:06,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:06,216 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:06,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/fea009ac6ea84d479f30dbf7e35cd865 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/fea009ac6ea84d479f30dbf7e35cd865 2023-04-07 01:54:06,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:06,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:06,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/fea009ac6ea84d479f30dbf7e35cd865, entries=200, sequenceid=618, filesize=14.3 K 2023-04-07 01:54:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:06,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:06,226 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/a3cade0637e74c27928c3a9a762530b3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a3cade0637e74c27928c3a9a762530b3 2023-04-07 01:54:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:06,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:06,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:06,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:06,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:06,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a3cade0637e74c27928c3a9a762530b3, entries=150, sequenceid=618, filesize=11.9 K 2023-04-07 01:54:06,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/f6edaabb52b748e99e3d98abedb10e6e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/f6edaabb52b748e99e3d98abedb10e6e 2023-04-07 01:54:06,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/f6edaabb52b748e99e3d98abedb10e6e, entries=150, sequenceid=618, filesize=11.9 K 2023-04-07 01:54:06,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e2fe8c1647b59c84578d50aa3cf4039c in 884ms, sequenceid=618, compaction requested=false 2023-04-07 01:54:06,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:06,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:06,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:06,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:06,335 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:06,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:06,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:06,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:06,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,338 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,338 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:06,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:06,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:06,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:06,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:06,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:06,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:06,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:06,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:06,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:06,339 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:06,339 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. started... 2023-04-07 01:54:06,340 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:06,340 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:06,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:06,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. as already flushing 2023-04-07 01:54:06,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:06,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832506597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:06,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832506700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:06,749 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/6f74520617ad4a65b8bdfd05b5e44df4 2023-04-07 01:54:06,771 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/25384588cc7844e1bd65957ddb08cd5e 2023-04-07 01:54:06,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:06,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832506902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,187 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dc8d1538c22a410b9ceb45cb03700e8b 2023-04-07 01:54:07,192 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/6f74520617ad4a65b8bdfd05b5e44df4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6f74520617ad4a65b8bdfd05b5e44df4 2023-04-07 01:54:07,197 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6f74520617ad4a65b8bdfd05b5e44df4, entries=150, sequenceid=653, filesize=11.9 K 2023-04-07 01:54:07,198 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/25384588cc7844e1bd65957ddb08cd5e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/25384588cc7844e1bd65957ddb08cd5e 2023-04-07 01:54:07,203 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/25384588cc7844e1bd65957ddb08cd5e, entries=150, sequenceid=653, filesize=11.9 K 2023-04-07 01:54:07,204 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/dc8d1538c22a410b9ceb45cb03700e8b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dc8d1538c22a410b9ceb45cb03700e8b 2023-04-07 01:54:07,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:07,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38436 deadline: 1680832507206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,210 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dc8d1538c22a410b9ceb45cb03700e8b, entries=150, sequenceid=653, filesize=11.9 K 2023-04-07 01:54:07,211 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e2fe8c1647b59c84578d50aa3cf4039c in 871ms, sequenceid=653, compaction requested=true 2023-04-07 01:54:07,211 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:07,211 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:07,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:07,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:07,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:07,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654481ea[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:07,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:07,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:07,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:07,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:07,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:07,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:07,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:07,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:07,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:07,221 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:07,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:07,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:07,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:07,221 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:07,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:07,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:07,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:07,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:07,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:07,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:07,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:07,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:07,233 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:07,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:07,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:07,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:07,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:07,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:07,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:07,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:07,233 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:07,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:07,275 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:61971 2023-04-07 01:54:07,275 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,278 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:61971 2023-04-07 01:54:07,278 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,278 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:61971 2023-04-07 01:54:07,278 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,284 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:61971 2023-04-07 01:54:07,284 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:07,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:07,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:07,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:54:07,554 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:61971 2023-04-07 01:54:07,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:07,555 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:07,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:07,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:07,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:07,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:07,560 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:61971 2023-04-07 01:54:07,560 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,710 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:61971 2023-04-07 01:54:07,710 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:07,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/d655cbc9f6594f8390abfc9588292ab0 2023-04-07 01:54:08,095 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:61971 2023-04-07 01:54:08,095 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:08,142 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:61971 2023-04-07 01:54:08,142 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 261 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4411 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4420 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1951 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5849 rows 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1939 2023-04-07 01:54:08,142 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5813 rows 2023-04-07 01:54:08,143 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:54:08,143 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:61971 2023-04-07 01:54:08,143 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:08,147 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:54:08,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:54:08,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:08,162 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832448161"}]},"ts":"1680832448161"} 2023-04-07 01:54:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:08,163 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:54:08,165 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:54:08,167 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, UNASSIGN}] 2023-04-07 01:54:08,168 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, UNASSIGN 2023-04-07 01:54:08,169 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e2fe8c1647b59c84578d50aa3cf4039c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:08,169 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832448169"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832448169"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832448169"}]},"ts":"1680832448169"} 2023-04-07 01:54:08,171 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:08,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:08,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:08,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e2fe8c1647b59c84578d50aa3cf4039c, disabling compactions & flushes 2023-04-07 01:54:08,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:08,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ee0c48480e124ef7b6fff7f23e4d090f 2023-04-07 01:54:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:08,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:08,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/a16e2eb2075441f8a5a3bddeb41a64b5 2023-04-07 01:54:08,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/d655cbc9f6594f8390abfc9588292ab0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d655cbc9f6594f8390abfc9588292ab0 2023-04-07 01:54:08,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d655cbc9f6594f8390abfc9588292ab0, entries=100, sequenceid=665, filesize=9.5 K 2023-04-07 01:54:08,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/ee0c48480e124ef7b6fff7f23e4d090f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ee0c48480e124ef7b6fff7f23e4d090f 2023-04-07 01:54:08,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ee0c48480e124ef7b6fff7f23e4d090f, entries=100, sequenceid=665, filesize=9.5 K 2023-04-07 01:54:08,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/a16e2eb2075441f8a5a3bddeb41a64b5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a16e2eb2075441f8a5a3bddeb41a64b5 2023-04-07 01:54:08,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a16e2eb2075441f8a5a3bddeb41a64b5, entries=100, sequenceid=665, filesize=9.5 K 2023-04-07 01:54:08,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for e2fe8c1647b59c84578d50aa3cf4039c in 1252ms, sequenceid=665, compaction requested=true 2023-04-07 01:54:08,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:08,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:08,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:08,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:08,807 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. because compaction request was cancelled 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. after waiting 0 ms 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:08,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:08,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. because compaction request was cancelled 2023-04-07 01:54:08,807 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. because compaction request was cancelled 2023-04-07 01:54:08,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:08,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e2fe8c1647b59c84578d50aa3cf4039c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=A 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=B 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e2fe8c1647b59c84578d50aa3cf4039c, store=C 2023-04-07 01:54:08,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:09,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/172e5289b9f24d6287dbcf2e674bc512 2023-04-07 01:54:09,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:09,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/d1a0895a85df4ea59537fbd44b3d4d5f 2023-04-07 01:54:10,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/4519245b04a541c590c4301d1bbfc0cf 2023-04-07 01:54:10,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/A/172e5289b9f24d6287dbcf2e674bc512 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/172e5289b9f24d6287dbcf2e674bc512 2023-04-07 01:54:10,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/172e5289b9f24d6287dbcf2e674bc512, entries=150, sequenceid=672, filesize=11.9 K 2023-04-07 01:54:10,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/B/d1a0895a85df4ea59537fbd44b3d4d5f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d1a0895a85df4ea59537fbd44b3d4d5f 2023-04-07 01:54:10,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d1a0895a85df4ea59537fbd44b3d4d5f, entries=150, sequenceid=672, filesize=11.9 K 2023-04-07 01:54:10,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/.tmp/C/4519245b04a541c590c4301d1bbfc0cf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4519245b04a541c590c4301d1bbfc0cf 2023-04-07 01:54:10,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4519245b04a541c590c4301d1bbfc0cf, entries=150, sequenceid=672, filesize=11.9 K 2023-04-07 01:54:10,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e2fe8c1647b59c84578d50aa3cf4039c in 1249ms, sequenceid=672, compaction requested=true 2023-04-07 01:54:10,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/88bb288eba544aef9884e215ddc81ab4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/11559caf4382417e9c158cd0257265df, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/b32613870a7c47a0b044dc1ba45e191d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/16e3d1448f4c48858f867866aeaba105, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6b774e0d04ae488da7ee7fb3c568a03e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/77374d28aa1b4d9ea0bd8bf5893b762f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/5c62cc1963ff46f097c7a9530efe0262, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c8913a3f1f6245d6b605e58fe657876c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8] to archive 2023-04-07 01:54:10,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:10,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/eae1622b2da444fcb3e5c35317841946 2023-04-07 01:54:10,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4fb4b31cd52e4526b7c1e2953ef1bf83 2023-04-07 01:54:10,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e9220123b5c9495aa70f4a9d96024f24 2023-04-07 01:54:10,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/88bb288eba544aef9884e215ddc81ab4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/88bb288eba544aef9884e215ddc81ab4 2023-04-07 01:54:10,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2acb45dca0c0421380dc580c89bed371 2023-04-07 01:54:10,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2036be2c74db4a41bfa9f6656c62d963 2023-04-07 01:54:10,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/30788cc88752493f80e109d30ad43596 2023-04-07 01:54:10,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/11559caf4382417e9c158cd0257265df to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/11559caf4382417e9c158cd0257265df 2023-04-07 01:54:10,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/725ea0c3e2a44df8bd1c12b51444a9e4 2023-04-07 01:54:10,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/4f63fb0e25db4fbe8bd31a6a802dc153 2023-04-07 01:54:10,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/73566c1c2f204e5dbb4776cd1b5031fd 2023-04-07 01:54:10,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/b32613870a7c47a0b044dc1ba45e191d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/b32613870a7c47a0b044dc1ba45e191d 2023-04-07 01:54:10,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/a10cf79358dd492ca281eb03cb757c28 2023-04-07 01:54:10,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9fba0d643b7f43fba230a54f42e35055 2023-04-07 01:54:10,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/af8520c56332413fafbdce8dea69bdaa 2023-04-07 01:54:10,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/16e3d1448f4c48858f867866aeaba105 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/16e3d1448f4c48858f867866aeaba105 2023-04-07 01:54:10,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/24fbc314d70a4cf985036fdf5a18fcc1 2023-04-07 01:54:10,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6b774e0d04ae488da7ee7fb3c568a03e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6b774e0d04ae488da7ee7fb3c568a03e 2023-04-07 01:54:10,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/79fa55421e9f49de8c64d2029b4e0871 2023-04-07 01:54:10,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c0ec253ef0b44d158a00a8983688501e 2023-04-07 01:54:10,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/3e5c296999e2448bb5a3b10bd2e843bf 2023-04-07 01:54:10,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/77374d28aa1b4d9ea0bd8bf5893b762f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/77374d28aa1b4d9ea0bd8bf5893b762f 2023-04-07 01:54:10,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/aaf244424ca745db913cd4b3f4e95d31 2023-04-07 01:54:10,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/1f368c858eca489da55de56ba439c848 2023-04-07 01:54:10,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d913351408e54d0d9ab8e782350f2866 2023-04-07 01:54:10,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/5c62cc1963ff46f097c7a9530efe0262 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/5c62cc1963ff46f097c7a9530efe0262 2023-04-07 01:54:10,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/da3d5e27bb304673acf5fba8b27ae727 2023-04-07 01:54:10,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/66e8eff59b2242dd956e92ccae777bf2 2023-04-07 01:54:10,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c2485608d1824bb2b4afe384a0f9a3ec 2023-04-07 01:54:10,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c8913a3f1f6245d6b605e58fe657876c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/c8913a3f1f6245d6b605e58fe657876c 2023-04-07 01:54:10,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/2141a8af04ea4e37aa814d96c745e650 2023-04-07 01:54:10,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/e3c07426255444daa72399b9a5fde221 2023-04-07 01:54:10,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9d63f9e9cc49435fb3d053a09d0596df 2023-04-07 01:54:10,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/7e741a1a1e774721abeaba45f7d192e8 2023-04-07 01:54:10,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/72723068a60f44758bbb2c49efc05d7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3e246d43ce9e4d428ea9a05be5f1fc06, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1b59c4f0076448ce9149bf761b8a482f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e42ff8fb78c44ea85b993976c7f5791, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/47fc946b8d8d4396bbe1b04718bb0691, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/bd0efe8648344612a3e3451e8a0e5937, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d4c3a6eaa174407589134df4ce31fecd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/f8af63d20e084cefa45fe1357e57478b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679] to archive 2023-04-07 01:54:10,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:10,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/65ba28173584436b8112d431e4cebd76 2023-04-07 01:54:10,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e56339d0e0f447fd9bd088f33d90a12f 2023-04-07 01:54:10,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d00f989b56ce4d5890382f07281e67ed 2023-04-07 01:54:10,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/72723068a60f44758bbb2c49efc05d7c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/72723068a60f44758bbb2c49efc05d7c 2023-04-07 01:54:10,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e99c3d816664145b295e72ae0aa0f52 2023-04-07 01:54:10,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/67a80c7d00c74e30b01a0656b3839aa9 2023-04-07 01:54:10,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0ad245e2ca47abbcaf4de691e387fa 2023-04-07 01:54:10,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3e246d43ce9e4d428ea9a05be5f1fc06 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3e246d43ce9e4d428ea9a05be5f1fc06 2023-04-07 01:54:10,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/3da1932af7034f309d5e0d1c86763b6c 2023-04-07 01:54:10,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8a1883e22bef4ae783d76463b9f92ab0 2023-04-07 01:54:10,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1b59c4f0076448ce9149bf761b8a482f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1b59c4f0076448ce9149bf761b8a482f 2023-04-07 01:54:10,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/55e4b92a8d574b97ab84583c4ce9221f 2023-04-07 01:54:10,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/aac3cc258e8c4f35896660eb55fe5146 2023-04-07 01:54:10,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6bb850e661c145c7a1a1e61b6c378a94 2023-04-07 01:54:10,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e42ff8fb78c44ea85b993976c7f5791 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e42ff8fb78c44ea85b993976c7f5791 2023-04-07 01:54:10,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/cca0a44cd4524c2d9d1b58864b84a21d 2023-04-07 01:54:10,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/283c60faec1a4116b30cb61d21b2b255 2023-04-07 01:54:10,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/47fc946b8d8d4396bbe1b04718bb0691 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/47fc946b8d8d4396bbe1b04718bb0691 2023-04-07 01:54:10,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/08f747420db948578d7941cdd850f740 2023-04-07 01:54:10,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/8e557c3c238f48149c09dc939515d784 2023-04-07 01:54:10,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/bd0efe8648344612a3e3451e8a0e5937 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/bd0efe8648344612a3e3451e8a0e5937 2023-04-07 01:54:10,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/2ac7a2cd4469472eb479f6c0a971fe91 2023-04-07 01:54:10,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/b9ed66c7b74048439d3238411c3aeace 2023-04-07 01:54:10,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/6e4060d78aa74c6993a276da793d4ffe 2023-04-07 01:54:10,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d4c3a6eaa174407589134df4ce31fecd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d4c3a6eaa174407589134df4ce31fecd 2023-04-07 01:54:10,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ce0717792d264b4fa4a9180a80662d3d 2023-04-07 01:54:10,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a438adc5f81a405abfef53b11e2682ee 2023-04-07 01:54:10,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/7464ee1193cc4c999ed790c46177148f 2023-04-07 01:54:10,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ed6a4d341ebd4d1a9e2f71fbe13c2139 2023-04-07 01:54:10,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/f8af63d20e084cefa45fe1357e57478b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/f8af63d20e084cefa45fe1357e57478b 2023-04-07 01:54:10,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/e4b2d0538e96402e9ddf5b8c6d8045a1 2023-04-07 01:54:10,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/40b9778b6c024c56a0919588a4345e10 2023-04-07 01:54:10,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/4f3e6a07a2ed41ffa5ab8e161ac4f53d 2023-04-07 01:54:10,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/1f073f949ac94067abcfaa71086d5679 2023-04-07 01:54:10,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/655d6e697b71490485134306d7d9ce04, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/6c87601642794c8283960d1aedf4e793, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/bfecef37ce1e48dfbdc59cb4e24a61e3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a1975325ac3b4388a73ece3a60e3a4ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4c900183036843fca5e88f290b65a075, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/913f7c64c5bd46eebdf4ff248c7a8ea3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/917b39a67837449693d8d998c0824548, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/8c5fe4daa0e94e53bc75f6e98d960e2d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4] to archive 2023-04-07 01:54:10,200 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:10,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ed17282226e14781b2d0d2b6316dc6f7 2023-04-07 01:54:10,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a7be4c3f145543b9a99dee42197208cc 2023-04-07 01:54:10,204 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/aa56f9607bd1437b868f8aa3b348695f 2023-04-07 01:54:10,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/655d6e697b71490485134306d7d9ce04 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/655d6e697b71490485134306d7d9ce04 2023-04-07 01:54:10,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7e047054e7114256bb6674cc4c957392 2023-04-07 01:54:10,207 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4623d26d63f740d9adb927a542b69fb4 2023-04-07 01:54:10,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/ef552b22118e495fb3343a1d0b671de5 2023-04-07 01:54:10,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/6c87601642794c8283960d1aedf4e793 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/6c87601642794c8283960d1aedf4e793 2023-04-07 01:54:10,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/456ea6e1587e48c28c843d62f947bbc7 2023-04-07 01:54:10,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e68ac8c11eae47af89d756e10cfcafb3 2023-04-07 01:54:10,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/bfecef37ce1e48dfbdc59cb4e24a61e3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/bfecef37ce1e48dfbdc59cb4e24a61e3 2023-04-07 01:54:10,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/43e7f54812194461a484fa95decfe248 2023-04-07 01:54:10,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/c09d3d4ae54d4cb79080f661f8ed5066 2023-04-07 01:54:10,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2755775620c444deacb5619c6c2dd287 2023-04-07 01:54:10,216 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a1975325ac3b4388a73ece3a60e3a4ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a1975325ac3b4388a73ece3a60e3a4ec 2023-04-07 01:54:10,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/3306e0accc5c46d0a56cdb2e5f290b9e 2023-04-07 01:54:10,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/99c3e54f83c94885857aa156dfb2e0ff 2023-04-07 01:54:10,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4c900183036843fca5e88f290b65a075 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4c900183036843fca5e88f290b65a075 2023-04-07 01:54:10,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5ffbb9f242b84b9881f88147b7cc5ad8 2023-04-07 01:54:10,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/2e2fbd858c9e4b3aa7d3bd2382e1a24a 2023-04-07 01:54:10,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/913f7c64c5bd46eebdf4ff248c7a8ea3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/913f7c64c5bd46eebdf4ff248c7a8ea3 2023-04-07 01:54:10,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/e5a7bcd21b1646d380f261b8a3e67a9e 2023-04-07 01:54:10,224 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/0705225b15e045d2a5a59358e30fdb29 2023-04-07 01:54:10,225 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/79517683c2974d6abf0377e4b5a7456f 2023-04-07 01:54:10,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/917b39a67837449693d8d998c0824548 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/917b39a67837449693d8d998c0824548 2023-04-07 01:54:10,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dff5ba3d49b34a7087f3a1067c8b7ab2 2023-04-07 01:54:10,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/41b9b34af5884616aca405f9487fadb7 2023-04-07 01:54:10,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/30c5892bc09c4c16a2ee3e92ff0dd209 2023-04-07 01:54:10,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/b26b7b47f0f14d9c8d614bf7823da49d 2023-04-07 01:54:10,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/8c5fe4daa0e94e53bc75f6e98d960e2d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/8c5fe4daa0e94e53bc75f6e98d960e2d 2023-04-07 01:54:10,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/063c620eb2b245e0a1a14eddb2421b81 2023-04-07 01:54:10,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dd8cc2b9f7d44b8ab46f62b23c063947 2023-04-07 01:54:10,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/7ce244d898104482a20281f12751312e 2023-04-07 01:54:10,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dacacc40dc614c5bbc66b7644391bbf4 2023-04-07 01:54:10,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/recovered.edits/675.seqid, newMaxSeqId=675, maxSeqId=1 2023-04-07 01:54:10,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c. 2023-04-07 01:54:10,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e2fe8c1647b59c84578d50aa3cf4039c: 2023-04-07 01:54:10,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:10,245 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e2fe8c1647b59c84578d50aa3cf4039c, regionState=CLOSED 2023-04-07 01:54:10,245 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832450245"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832450245"}]},"ts":"1680832450245"} 2023-04-07 01:54:10,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-07 01:54:10,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure e2fe8c1647b59c84578d50aa3cf4039c, server=jenkins-hbase4.apache.org,44961,1680832421898 in 2.0760 sec 2023-04-07 01:54:10,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-07 01:54:10,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e2fe8c1647b59c84578d50aa3cf4039c, UNASSIGN in 2.0810 sec 2023-04-07 01:54:10,251 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832450250"}]},"ts":"1680832450250"} 2023-04-07 01:54:10,252 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:54:10,253 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:54:10,255 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1000 sec 2023-04-07 01:54:10,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 01:54:10,267 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-07 01:54:10,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:54:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,279 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,281 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-07 01:54:10,285 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:10,288 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/recovered.edits] 2023-04-07 01:54:10,291 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/172e5289b9f24d6287dbcf2e674bc512 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/172e5289b9f24d6287dbcf2e674bc512 2023-04-07 01:54:10,292 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6f74520617ad4a65b8bdfd05b5e44df4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/6f74520617ad4a65b8bdfd05b5e44df4 2023-04-07 01:54:10,294 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9b9e47d9a74243b9938feb5d70c90674 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/9b9e47d9a74243b9938feb5d70c90674 2023-04-07 01:54:10,295 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d655cbc9f6594f8390abfc9588292ab0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/d655cbc9f6594f8390abfc9588292ab0 2023-04-07 01:54:10,296 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/fea009ac6ea84d479f30dbf7e35cd865 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/A/fea009ac6ea84d479f30dbf7e35cd865 2023-04-07 01:54:10,299 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/25384588cc7844e1bd65957ddb08cd5e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/25384588cc7844e1bd65957ddb08cd5e 2023-04-07 01:54:10,300 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a3cade0637e74c27928c3a9a762530b3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a3cade0637e74c27928c3a9a762530b3 2023-04-07 01:54:10,302 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a4b85046360f4ca1b0b4ef17c8a0a15f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/a4b85046360f4ca1b0b4ef17c8a0a15f 2023-04-07 01:54:10,303 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d1a0895a85df4ea59537fbd44b3d4d5f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/d1a0895a85df4ea59537fbd44b3d4d5f 2023-04-07 01:54:10,304 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ee0c48480e124ef7b6fff7f23e4d090f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/B/ee0c48480e124ef7b6fff7f23e4d090f 2023-04-07 01:54:10,307 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4519245b04a541c590c4301d1bbfc0cf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/4519245b04a541c590c4301d1bbfc0cf 2023-04-07 01:54:10,308 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5031f38ec858405d8f3a0a505a3acd75 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/5031f38ec858405d8f3a0a505a3acd75 2023-04-07 01:54:10,309 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a16e2eb2075441f8a5a3bddeb41a64b5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/a16e2eb2075441f8a5a3bddeb41a64b5 2023-04-07 01:54:10,310 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dc8d1538c22a410b9ceb45cb03700e8b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/dc8d1538c22a410b9ceb45cb03700e8b 2023-04-07 01:54:10,312 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/f6edaabb52b748e99e3d98abedb10e6e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/C/f6edaabb52b748e99e3d98abedb10e6e 2023-04-07 01:54:10,314 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/recovered.edits/675.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c/recovered.edits/675.seqid 2023-04-07 01:54:10,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/e2fe8c1647b59c84578d50aa3cf4039c 2023-04-07 01:54:10,315 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:54:10,317 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,339 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:54:10,383 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:54:10,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-07 01:54:10,384 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,384 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:54:10,384 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832450384"}]},"ts":"9223372036854775807"} 2023-04-07 01:54:10,387 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:54:10,387 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e2fe8c1647b59c84578d50aa3cf4039c, NAME => 'TestAcidGuarantees,,1680832426543.e2fe8c1647b59c84578d50aa3cf4039c.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:54:10,387 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:54:10,387 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832450387"}]},"ts":"9223372036854775807"} 2023-04-07 01:54:10,389 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:54:10,391 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,392 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 119 msec 2023-04-07 01:54:10,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-07 01:54:10,584 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-07 01:54:10,601 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 205) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x33c18644-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1874561944_15 at /127.0.0.1:47576 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1874561944_15 at /127.0.0.1:47626 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x33c18644-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x33c18644-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x33c18644-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1874561944_15 at /127.0.0.1:47682 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:44961-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=568 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=398 (was 400), ProcessCount=169 (was 169), AvailableMemoryMB=2891 (was 3493) 2023-04-07 01:54:10,612 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=568, MaxFileDescriptor=60000, SystemLoadAverage=398, ProcessCount=169, AvailableMemoryMB=2891 2023-04-07 01:54:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:54:10,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:54:10,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:10,617 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:54:10,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-07 01:54:10,618 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:54:10,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:10,619 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:10,620 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 empty. 2023-04-07 01:54:10,620 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:10,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:54:10,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:10,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:11,029 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:54:11,030 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9bfd0d326e03d54dad8a7c3526a41584, NAME => 'TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:54:11,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9bfd0d326e03d54dad8a7c3526a41584, disabling compactions & flushes 2023-04-07 01:54:11,437 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. after waiting 0 ms 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,437 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,437 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:11,439 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:54:11,441 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832451440"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832451440"}]},"ts":"1680832451440"} 2023-04-07 01:54:11,442 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:54:11,443 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:54:11,443 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832451443"}]},"ts":"1680832451443"} 2023-04-07 01:54:11,444 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:54:11,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, ASSIGN}] 2023-04-07 01:54:11,450 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, ASSIGN 2023-04-07 01:54:11,450 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:54:11,602 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:11,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832451602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832451602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832451602"}]},"ts":"1680832451602"} 2023-04-07 01:54:11,604 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:11,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:11,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9bfd0d326e03d54dad8a7c3526a41584, NAME => 'TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:54:11,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:54:11,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,761 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,763 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A 2023-04-07 01:54:11,763 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A 2023-04-07 01:54:11,763 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:11,763 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName A 2023-04-07 01:54:11,764 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:11,764 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,765 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B 2023-04-07 01:54:11,765 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B 2023-04-07 01:54:11,765 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:11,765 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName B 2023-04-07 01:54:11,766 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:11,766 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,767 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C 2023-04-07 01:54:11,767 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C 2023-04-07 01:54:11,767 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:11,767 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName C 2023-04-07 01:54:11,768 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:11,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:54:11,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:11,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:54:11,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9bfd0d326e03d54dad8a7c3526a41584; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61287561, jitterRate=-0.08674417436122894}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:54:11,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:11,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., pid=18, masterSystemTime=1680832451755 2023-04-07 01:54:11,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:11,776 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:11,777 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832451776"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832451776"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832451776"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832451776"}]},"ts":"1680832451776"} 2023-04-07 01:54:11,779 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-07 01:54:11,780 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 in 175 msec 2023-04-07 01:54:11,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-07 01:54:11,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, ASSIGN in 332 msec 2023-04-07 01:54:11,782 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:54:11,782 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832451782"}]},"ts":"1680832451782"} 2023-04-07 01:54:11,783 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:54:11,786 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:54:11,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-07 01:54:12,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 01:54:12,723 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-07 01:54:12,724 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:12,730 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:12,732 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:12,736 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:12,738 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:54:12,740 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:54:12,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:54:12,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:54:12,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:13,173 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:54:13,175 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:54:13,175 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:54:13,179 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 01:54:13,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, REOPEN/MOVE}] 2023-04-07 01:54:13,192 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, REOPEN/MOVE 2023-04-07 01:54:13,193 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,193 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832453193"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832453193"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832453193"}]},"ts":"1680832453193"} 2023-04-07 01:54:13,194 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:13,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9bfd0d326e03d54dad8a7c3526a41584, disabling compactions & flushes 2023-04-07 01:54:13,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. after waiting 0 ms 2023-04-07 01:54:13,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 01:54:13,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:13,351 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9bfd0d326e03d54dad8a7c3526a41584 to self. 2023-04-07 01:54:13,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,353 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=CLOSED 2023-04-07 01:54:13,354 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832453353"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832453353"}]},"ts":"1680832453353"} 2023-04-07 01:54:13,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-07 01:54:13,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 in 161 msec 2023-04-07 01:54:13,358 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=true 2023-04-07 01:54:13,508 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,508 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832453508"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832453508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832453508"}]},"ts":"1680832453508"} 2023-04-07 01:54:13,510 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:13,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9bfd0d326e03d54dad8a7c3526a41584, NAME => 'TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:54:13,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:54:13,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,669 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,670 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A 2023-04-07 01:54:13,670 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A 2023-04-07 01:54:13,670 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:13,673 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName A 2023-04-07 01:54:13,675 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:13,675 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,677 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B 2023-04-07 01:54:13,677 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B 2023-04-07 01:54:13,677 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:13,677 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName B 2023-04-07 01:54:13,678 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:13,678 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,679 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C 2023-04-07 01:54:13,679 DEBUG [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C 2023-04-07 01:54:13,679 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:13,679 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bfd0d326e03d54dad8a7c3526a41584 columnFamilyName C 2023-04-07 01:54:13,680 INFO [StoreOpener-9bfd0d326e03d54dad8a7c3526a41584-1] regionserver.HStore(310): Store=9bfd0d326e03d54dad8a7c3526a41584/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:13,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:54:13,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9bfd0d326e03d54dad8a7c3526a41584; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75059554, jitterRate=0.11847451329231262}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:54:13,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:13,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., pid=23, masterSystemTime=1680832453662 2023-04-07 01:54:13,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,689 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,689 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832453689"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832453689"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832453689"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832453689"}]},"ts":"1680832453689"} 2023-04-07 01:54:13,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-07 01:54:13,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 in 180 msec 2023-04-07 01:54:13,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-07 01:54:13,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, REOPEN/MOVE in 501 msec 2023-04-07 01:54:13,696 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-07 01:54:13,697 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 517 msec 2023-04-07 01:54:13,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 944 msec 2023-04-07 01:54:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-07 01:54:13,707 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,712 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,713 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,716 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,717 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,722 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,723 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,730 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,730 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,735 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,735 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,741 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,741 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,748 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,748 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,753 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,753 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:13,757 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:13,763 DEBUG [hconnection-0x57cee27a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,763 DEBUG [hconnection-0x25ae36a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,764 DEBUG [hconnection-0x166dbc4a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,766 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:13,775 DEBUG [hconnection-0x12abfe3b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,776 DEBUG [hconnection-0x24abfeb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,777 DEBUG [hconnection-0x30652218-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,777 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,778 DEBUG [hconnection-0x49c5666b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,782 DEBUG [hconnection-0x479f41c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,783 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,783 DEBUG [hconnection-0x44715b92-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:13,785 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,785 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,786 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,787 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48632, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,789 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,789 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:13,799 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:13,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:13,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:13,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:13,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:13,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:13,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:13,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:13,803 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,803 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:13,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:13,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:13,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:13,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:13,811 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 01:54:13,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:13,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:13,813 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:13,814 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:13,814 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:13,814 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:13,814 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:13,814 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:13,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832453834, max timestamp=1680832453839Num uniques 859; 2023-04-07 01:54:13,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:13,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832453834, max timestamp=1680832453839Num uniques 859; 2023-04-07 01:54:13,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:13,846 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832453834, max timestamp=1680832453839Num uniques 859; 2023-04-07 01:54:13,846 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:13,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832513884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832513886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832513886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832513887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680832453872, max timestamp=1680832453885Num uniques 1018; 2023-04-07 01:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-07 01:54:13,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832453883, max timestamp=1680832453885Num uniques 1018; 2023-04-07 01:54:13,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:13,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832453883, max timestamp=1680832453885Num uniques 1018; 2023-04-07 01:54:13,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:13,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832513941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832513989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832513989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832513990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:13,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832513990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832514046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832514200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832514200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832514200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832514201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832514249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,299 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040774a9bf7a43044068baa64553c67e0332 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040774a9bf7a43044068baa64553c67e0332 2023-04-07 01:54:14,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832514503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832514504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832514504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832514504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:14,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832514552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,719 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/a134764e2f57445a80923763eb6abb5e 2023-04-07 01:54:14,747 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/5eaacb92e5bb40098f2d970beab63e93 2023-04-07 01:54:14,764 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/653b8c462fdf49fb9eae575a4550b905 2023-04-07 01:54:14,769 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/a134764e2f57445a80923763eb6abb5e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e 2023-04-07 01:54:14,774 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e, entries=150, sequenceid=14, filesize=25.1 K 2023-04-07 01:54:14,776 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/5eaacb92e5bb40098f2d970beab63e93 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93 2023-04-07 01:54:14,781 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93, entries=150, sequenceid=14, filesize=11.6 K 2023-04-07 01:54:14,782 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/653b8c462fdf49fb9eae575a4550b905 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905 2023-04-07 01:54:14,790 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905, entries=150, sequenceid=14, filesize=11.6 K 2023-04-07 01:54:14,791 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9bfd0d326e03d54dad8a7c3526a41584 in 980ms, sequenceid=14, compaction requested=false 2023-04-07 01:54:14,791 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:14,791 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:14,794 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625914f3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:14,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:14,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,798 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:14,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:14,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:14,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:14,801 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:14,805 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:14,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:14,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:14,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:14,805 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:14,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:14,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:14,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:14,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:14,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:14,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:14,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:14,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:14,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:14,814 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:14,814 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:14,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:14,914 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:14,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:14,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:14,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:14,919 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:14,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:14,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:14,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:14,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:14,921 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:14,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:14,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:14,921 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:14,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:14,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:14,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:14,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:14,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:14,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:14,923 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-07 01:54:14,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:14,924 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:14,924 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:14,924 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:14,924 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:14,924 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:14,964 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304079699dc975bdb4f4ab57bb380378aab2e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079699dc975bdb4f4ab57bb380378aab2e 2023-04-07 01:54:14,994 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=67, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/98632fe3309c4ca28103ba0929334723 2023-04-07 01:54:15,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:15,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:15,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832515058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,443 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/fface081e1f3406f8ae3bb4413f0e93a 2023-04-07 01:54:15,462 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/5d2ce0802f174ffdbb834dc35336d2e2 2023-04-07 01:54:15,469 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/98632fe3309c4ca28103ba0929334723 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723 2023-04-07 01:54:15,477 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723, entries=150, sequenceid=67, filesize=25.1 K 2023-04-07 01:54:15,478 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/fface081e1f3406f8ae3bb4413f0e93a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a 2023-04-07 01:54:15,504 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a, entries=150, sequenceid=67, filesize=11.6 K 2023-04-07 01:54:15,505 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/5d2ce0802f174ffdbb834dc35336d2e2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2 2023-04-07 01:54:15,510 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2, entries=150, sequenceid=67, filesize=11.6 K 2023-04-07 01:54:15,511 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 9bfd0d326e03d54dad8a7c3526a41584 in 588ms, sequenceid=67, compaction requested=false 2023-04-07 01:54:15,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:15,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:15,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:15,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:15,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:15,514 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:15,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d72f5c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:15,516 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:15,516 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:15,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,518 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:15,518 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:15,518 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:15,526 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:15,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:15,531 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:15,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:15,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:15,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:15,531 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:15,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,537 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,540 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,545 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:15,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,546 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:15,546 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:15,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:15,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:15,646 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:15,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:15,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:15,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832455643, max timestamp=1680832455655Num uniques 859; 2023-04-07 01:54:15,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:15,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832455643, max timestamp=1680832455655Num uniques 859; 2023-04-07 01:54:15,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:15,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832455643, max timestamp=1680832455655Num uniques 859; 2023-04-07 01:54:15,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:15,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:15,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:15,672 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:15,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:15,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:15,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:15,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:15,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:15,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,678 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:15,678 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:15,678 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:15,681 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:15,682 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304076aa63bd414b741cba269880ccf36896e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076aa63bd414b741cba269880ccf36896e 2023-04-07 01:54:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:15,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f75860[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:15,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:15,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:15,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:15,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:15,694 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:15,694 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:15,694 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:15,694 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:15,694 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:15,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,697 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,698 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:15,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:15,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,702 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,702 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,702 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:15,702 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:15,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:15,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,792 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:54:15,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,802 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:15,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:15,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:15,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:15,809 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:15,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:15,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:15,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,811 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:15,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:15,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:15,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:15,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:15,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:15,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:15,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:15,820 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,820 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:15,820 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:15,820 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:15,822 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:15,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567dd0a5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:15,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:15,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:15,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:15,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:15,834 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:15,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:15,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:15,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:15,834 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:15,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,838 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:15,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:15,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,844 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:15,844 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:15,944 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:15,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:15,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:15,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:15,948 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:15,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:15,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:15,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,951 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,951 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:15,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:15,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:15,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:15,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:15,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:15,953 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,953 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:15,953 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:15,953 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:15,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:15,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:15,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca9a815[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:15,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:15,957 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,957 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:15,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:15,961 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:15,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:15,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:15,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:15,961 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:15,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832515963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:15,964 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,966 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:15,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:15,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:15,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,969 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:15,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:15,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:15,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:15,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:15,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:15,969 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:15,969 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:15,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832515995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832515995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:15,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:15,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832515995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,069 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:16,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832516069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:16,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:16,073 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:16,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:16,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:16,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,075 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:16,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:16,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:16,077 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,077 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:16,077 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:16,077 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:16,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dbc513[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:16,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:16,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,086 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:16,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:16,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:16,086 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,086 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:16,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:16,091 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:16,091 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:16,091 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:16,091 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:16,091 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:16,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,097 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:16,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,101 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:16,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,101 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,118 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=78, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/d1d8d2641b9f455890a38339dec8251c 2023-04-07 01:54:16,201 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:16,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:16,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:16,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:16,208 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:16,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:16,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:16,209 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,209 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,210 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:16,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:16,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:16,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:16,212 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,212 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:16,212 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:16,212 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:16,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,215 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,215 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:16,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec2bd50[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:16,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:16,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:16,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:16,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:16,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:16,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:16,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:16,227 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:16,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:16,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:16,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:16,227 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:16,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,230 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,232 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,235 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:16,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:16,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,236 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:16,236 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:16,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832516264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832516299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832516299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832516302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,336 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:16,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:16,340 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:16,340 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,340 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:16,340 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:16,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:16,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:16,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:16,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:16,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:16,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:16,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:16,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:16,344 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,344 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:16,345 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:16,345 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,345 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:16,345 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:16,345 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,345 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:16,345 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:16,347 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,347 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,348 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,348 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:16,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3682acd5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:16,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:16,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,350 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:16,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:16,353 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:16,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,354 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:16,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:16,357 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:16,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:16,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:16,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:16,357 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:16,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,366 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:16,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,366 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,366 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:16,366 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:16,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:16,466 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:16,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:16,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:16,471 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:16,471 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,471 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:16,471 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:16,471 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,471 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,473 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:16,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:16,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:16,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:16,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:16,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:16,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:16,476 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,476 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:16,476 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:16,476 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:16,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:16,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:16,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a4310[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:16,481 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:16,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,483 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,483 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:16,483 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:16,483 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:16,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:16,486 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:16,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:16,489 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:16,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:16,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:16,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:16,490 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:16,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,492 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:16,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:16,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:16,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:16,499 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:16,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,500 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:16,500 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:16,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/7e0acd8f6a41404a96042131329e079e 2023-04-07 01:54:16,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/59553f2a1b8e47a790554d51fc567b55 2023-04-07 01:54:16,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/d1d8d2641b9f455890a38339dec8251c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c 2023-04-07 01:54:16,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c, entries=150, sequenceid=78, filesize=25.1 K 2023-04-07 01:54:16,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/7e0acd8f6a41404a96042131329e079e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e 2023-04-07 01:54:16,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e, entries=150, sequenceid=78, filesize=11.6 K 2023-04-07 01:54:16,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/59553f2a1b8e47a790554d51fc567b55 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55 2023-04-07 01:54:16,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55, entries=150, sequenceid=78, filesize=11.6 K 2023-04-07 01:54:16,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9bfd0d326e03d54dad8a7c3526a41584 in 945ms, sequenceid=78, compaction requested=true 2023-04-07 01:54:16,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:16,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:16,573 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:16,573 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:16,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:16,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:16,574 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:16,574 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:16,574 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,574 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=34.8 K 2023-04-07 01:54:16,574 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:16,575 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:16,575 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,575 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5eaacb92e5bb40098f2d970beab63e93, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832453802 2023-04-07 01:54:16,575 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=75.4 K 2023-04-07 01:54:16,575 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fface081e1f3406f8ae3bb4413f0e93a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680832453922 2023-04-07 01:54:16,575 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a134764e2f57445a80923763eb6abb5e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832453802 2023-04-07 01:54:16,576 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7e0acd8f6a41404a96042131329e079e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:16,576 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 98632fe3309c4ca28103ba0929334723, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680832453922 2023-04-07 01:54:16,577 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d1d8d2641b9f455890a38339dec8251c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:16,585 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#127 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:16,595 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#126 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:16,600 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:16,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:16,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:16,606 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/a1b44857127343108903f21c2f204607 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/a1b44857127343108903f21c2f204607 2023-04-07 01:54:16,612 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into a1b44857127343108903f21c2f204607(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:16,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:16,612 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=13, startTime=1680832456573; duration=0sec 2023-04-07 01:54:16,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:16,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:16,615 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:16,615 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:16,615 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,615 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=34.8 K 2023-04-07 01:54:16,616 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 653b8c462fdf49fb9eae575a4550b905, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832453802 2023-04-07 01:54:16,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:16,616 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5d2ce0802f174ffdbb834dc35336d2e2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680832453922 2023-04-07 01:54:16,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:16,616 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:16,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:16,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:16,617 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 59553f2a1b8e47a790554d51fc567b55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:16,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:16,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:16,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:16,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:16,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:16,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:16,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:16,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:16,624 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:16,624 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:16,624 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-07 01:54:16,624 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:16,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:16,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:16,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:16,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:16,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:16,633 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:16,664 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/ce2cea476a5c46e0a445baa5f5844aa0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/ce2cea476a5c46e0a445baa5f5844aa0 2023-04-07 01:54:16,672 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into ce2cea476a5c46e0a445baa5f5844aa0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:16,672 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:16,672 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=13, startTime=1680832456573; duration=0sec 2023-04-07 01:54:16,672 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:16,680 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/8bab34bac53442d48f82e406fc0f0160 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8bab34bac53442d48f82e406fc0f0160 2023-04-07 01:54:16,683 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040776164e828ada41b7a38bdf0b4ee4391a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040776164e828ada41b7a38bdf0b4ee4391a 2023-04-07 01:54:16,691 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into 8bab34bac53442d48f82e406fc0f0160(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:16,692 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:16,692 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=13, startTime=1680832456572; duration=0sec 2023-04-07 01:54:16,692 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:16,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:16,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:16,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832516802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832516805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832516805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832516806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832516909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:16,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:16,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832516909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,101 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/3a188bc986fa45d6ace3e631bf3cac87 2023-04-07 01:54:17,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832517114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832517114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832517417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832517417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,538 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/bc97ce6c155c48b3a39233718e45bc6f 2023-04-07 01:54:17,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832517811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832517814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832517920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832517924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,960 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/d0aff4b7e1bd4c28ab6d58da7e764498 2023-04-07 01:54:17,966 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/3a188bc986fa45d6ace3e631bf3cac87 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87 2023-04-07 01:54:17,971 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87, entries=150, sequenceid=113, filesize=25.1 K 2023-04-07 01:54:17,971 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/bc97ce6c155c48b3a39233718e45bc6f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f 2023-04-07 01:54:17,976 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f, entries=150, sequenceid=113, filesize=11.6 K 2023-04-07 01:54:17,976 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/d0aff4b7e1bd4c28ab6d58da7e764498 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498 2023-04-07 01:54:17,980 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498, entries=150, sequenceid=113, filesize=11.6 K 2023-04-07 01:54:17,981 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=60.38 KB/61830 for 9bfd0d326e03d54dad8a7c3526a41584 in 1357ms, sequenceid=113, compaction requested=false 2023-04-07 01:54:17,981 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:17,981 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:17,983 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:17,984 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,984 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:17,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:17,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad4edb5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:17,987 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:17,987 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:17,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:17,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:17,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:17,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:17,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:17,993 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:17,993 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:17,993 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:17,993 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:17,993 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:17,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:17,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:17,995 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:17,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:17,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:17,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:17,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:17,999 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:17,999 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:17,999 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:17,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:18,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:18,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,099 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:18,100 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407052619afc80047a8a8cd25302d887b13 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407052619afc80047a8a8cd25302d887b13 2023-04-07 01:54:18,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:18,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:18,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:18,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:18,103 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:18,103 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:18,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:18,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,105 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,106 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/885804b629b54a0f8d16174fe8135267 2023-04-07 01:54:18,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:18,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:18,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:18,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:18,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:18,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:18,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,110 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6167f782[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:18,114 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:18,114 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,116 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:18,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:18,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/6cdd772ab288483c9afbc8b48d16123d 2023-04-07 01:54:18,119 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,119 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,119 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,119 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:18,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:18,129 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:18,129 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:18,129 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:18,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,129 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:18,129 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:18,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832518128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,131 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,131 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,131 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,132 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,133 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,136 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:18,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:18,136 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,137 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832518230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,237 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:18,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:18,240 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:18,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:18,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:18,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,242 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:18,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,243 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:18,244 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,244 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:18,244 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:18,244 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:18,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:18,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a00b7f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:18,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:18,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,253 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,253 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:18,253 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:18,253 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:18,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:18,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:18,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:18,258 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:18,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:18,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:18,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:18,258 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:18,259 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,259 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,259 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,260 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,261 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:18,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:18,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,266 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:18,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,266 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:18,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,366 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:18,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:18,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:18,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:18,370 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:18,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:18,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:18,373 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,373 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:18,374 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:18,374 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:18,375 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,375 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,375 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,375 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173792f8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:18,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:18,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,379 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:18,379 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,379 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:18,380 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,380 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:18,383 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:18,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:18,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:18,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:18,383 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:18,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:18,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:18,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,390 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:18,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,390 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:18,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832518434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,490 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:18,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:18,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:18,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:18,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:18,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,496 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,497 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:18,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:18,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:18,498 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,499 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:18,499 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:18,499 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:18,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a536cee[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:18,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:18,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:18,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:18,505 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:18,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:18,509 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:18,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:18,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:18,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:18,509 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:18,512 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,512 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,512 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:18,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,512 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,517 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:18,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:18,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:18,517 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:18,517 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:18,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:18,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:18,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/d88a8844d3f3403590c098afe8b7225c 2023-04-07 01:54:18,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/885804b629b54a0f8d16174fe8135267 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267 2023-04-07 01:54:18,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267, entries=150, sequenceid=128, filesize=25.1 K 2023-04-07 01:54:18,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/6cdd772ab288483c9afbc8b48d16123d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d 2023-04-07 01:54:18,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d, entries=150, sequenceid=128, filesize=11.6 K 2023-04-07 01:54:18,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/d88a8844d3f3403590c098afe8b7225c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c 2023-04-07 01:54:18,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c, entries=150, sequenceid=128, filesize=11.6 K 2023-04-07 01:54:18,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9bfd0d326e03d54dad8a7c3526a41584 in 494ms, sequenceid=128, compaction requested=true 2023-04-07 01:54:18,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:18,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:18,574 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:18,574 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:18,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:18,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:18,575 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:18,575 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:18,575 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:18,575 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,575 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:18,575 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8bab34bac53442d48f82e406fc0f0160, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=75.5 K 2023-04-07 01:54:18,575 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,576 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/a1b44857127343108903f21c2f204607, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=34.9 K 2023-04-07 01:54:18,576 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8bab34bac53442d48f82e406fc0f0160, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:18,576 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a1b44857127343108903f21c2f204607, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:18,576 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3a188bc986fa45d6ace3e631bf3cac87, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832455675 2023-04-07 01:54:18,577 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bc97ce6c155c48b3a39233718e45bc6f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832455675 2023-04-07 01:54:18,577 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 885804b629b54a0f8d16174fe8135267, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:18,577 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6cdd772ab288483c9afbc8b48d16123d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:18,586 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:18,614 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#136 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:18,617 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:18,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:18,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:18,621 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/0d0667ad81ec45cb9f8e21d6b97e3021 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0d0667ad81ec45cb9f8e21d6b97e3021 2023-04-07 01:54:18,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:18,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:18,622 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:18,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:18,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:18,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:18,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:18,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:18,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:18,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:18,628 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into 0d0667ad81ec45cb9f8e21d6b97e3021(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:18,628 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:18,628 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=13, startTime=1680832458574; duration=0sec 2023-04-07 01:54:18,628 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:18,628 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:18,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:18,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:18,629 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:18,629 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:18,629 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:18,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:18,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:18,632 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:18,632 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:18,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:18,632 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:18,632 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/ce2cea476a5c46e0a445baa5f5844aa0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=34.9 K 2023-04-07 01:54:18,633 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:18,633 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ce2cea476a5c46e0a445baa5f5844aa0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680832455013 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:18,633 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:18,634 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d0aff4b7e1bd4c28ab6d58da7e764498, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832455675 2023-04-07 01:54:18,636 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d88a8844d3f3403590c098afe8b7225c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:18,649 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#138 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:18,657 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304077943e9183e8247bf8052dfd20e802231 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077943e9183e8247bf8052dfd20e802231 2023-04-07 01:54:18,662 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/30f699eece8c4de59d0eae48eada088f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30f699eece8c4de59d0eae48eada088f 2023-04-07 01:54:18,667 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 30f699eece8c4de59d0eae48eada088f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:18,667 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:18,667 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=13, startTime=1680832458574; duration=0sec 2023-04-07 01:54:18,667 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:18,674 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/96e8e3f3b1fe4829aa0223bd19a1e7b5 2023-04-07 01:54:18,687 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/65ec605634c24342967d2d290184303f 2023-04-07 01:54:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:18,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:18,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832518773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832518875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832518926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:18,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832518931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832519078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,121 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/4a16b2692ca347a3911d1062c44ca8b1 2023-04-07 01:54:19,126 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/96e8e3f3b1fe4829aa0223bd19a1e7b5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5 2023-04-07 01:54:19,130 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5, entries=150, sequenceid=148, filesize=25.3 K 2023-04-07 01:54:19,131 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/65ec605634c24342967d2d290184303f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f 2023-04-07 01:54:19,137 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f, entries=150, sequenceid=148, filesize=11.8 K 2023-04-07 01:54:19,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/4a16b2692ca347a3911d1062c44ca8b1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1 2023-04-07 01:54:19,142 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1, entries=150, sequenceid=148, filesize=11.8 K 2023-04-07 01:54:19,142 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9bfd0d326e03d54dad8a7c3526a41584 in 510ms, sequenceid=148, compaction requested=false 2023-04-07 01:54:19,143 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:19,143 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:19,145 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145649c1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:19,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:19,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,149 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:19,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:19,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:19,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:19,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:19,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:19,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:19,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:19,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:19,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:19,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:19,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:19,159 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:19,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:19,163 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:19,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:19,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:19,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:19,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:19,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:19,164 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:19,164 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:19,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,264 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:19,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:19,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:19,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:19,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:19,267 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:19,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:19,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:19,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:19,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:19,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,269 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:19,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:19,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:19,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:19,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:19,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:19,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:19,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:19,271 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:19,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:19,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:19,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:19,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832519425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832519529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,690 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407acfa3c914052477294d32125b66dac07 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407acfa3c914052477294d32125b66dac07 2023-04-07 01:54:19,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832519731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832519813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,816 DEBUG [Thread-909] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832519833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:19,836 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:19,850 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/56e6b508c8cb47edb9f86371f54b6880 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/56e6b508c8cb47edb9f86371f54b6880 2023-04-07 01:54:19,855 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into 56e6b508c8cb47edb9f86371f54b6880(size=25.3 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-07 01:54:19,855 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:19,855 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=13, startTime=1680832458574; duration=1sec 2023-04-07 01:54:19,855 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:20,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:20,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832520033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,099 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=162, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/9822d6543cdc4d9da50da471acdbaa3b 2023-04-07 01:54:20,511 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/47179463e513404f8a4cf0801b520a5e 2023-04-07 01:54:20,531 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/0ee121acb51b4113879324dafb26b152 2023-04-07 01:54:20,542 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/9822d6543cdc4d9da50da471acdbaa3b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b 2023-04-07 01:54:20,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:20,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832520540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,550 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b, entries=150, sequenceid=162, filesize=25.3 K 2023-04-07 01:54:20,551 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/47179463e513404f8a4cf0801b520a5e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e 2023-04-07 01:54:20,559 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 01:54:20,560 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/0ee121acb51b4113879324dafb26b152 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152 2023-04-07 01:54:20,566 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 01:54:20,566 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9bfd0d326e03d54dad8a7c3526a41584 in 1295ms, sequenceid=162, compaction requested=true 2023-04-07 01:54:20,566 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:20,567 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:20,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566f43c7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:20,581 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:20,581 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:20,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:20,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:20,588 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:20,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:20,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:20,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:20,588 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:20,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:20,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:20,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:20,590 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:20,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:20,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:20,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:20,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:20,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:20,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:20,595 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:20,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,595 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:20,595 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:20,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:20,695 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:20,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:20,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:20,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:20,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:20,699 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:20,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:20,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:20,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:20,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:20,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:20,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:20,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:20,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:20,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:20,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:20,703 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:20,732 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407eeccb76e27294dbeafaee0e3045cab99 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eeccb76e27294dbeafaee0e3045cab99 2023-04-07 01:54:20,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:20,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832520968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832520969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832521077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832521077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,139 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/97cde93c85934d96ac564778773824e2 2023-04-07 01:54:21,162 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/0b5481465e5642308082d21777322fa3 2023-04-07 01:54:21,191 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/13e8a8d1629943b89e63f6b8777b2b88 2023-04-07 01:54:21,197 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/97cde93c85934d96ac564778773824e2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2 2023-04-07 01:54:21,203 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2, entries=150, sequenceid=182, filesize=25.3 K 2023-04-07 01:54:21,204 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/0b5481465e5642308082d21777322fa3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3 2023-04-07 01:54:21,209 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3, entries=150, sequenceid=182, filesize=11.8 K 2023-04-07 01:54:21,210 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/13e8a8d1629943b89e63f6b8777b2b88 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88 2023-04-07 01:54:21,214 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88, entries=150, sequenceid=182, filesize=11.8 K 2023-04-07 01:54:21,215 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9bfd0d326e03d54dad8a7c3526a41584 in 512ms, sequenceid=182, compaction requested=true 2023-04-07 01:54:21,215 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:21,215 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d64aba0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,228 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,228 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,233 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,239 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,239 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,239 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,240 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:21,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:54:21,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:21,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:21,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:21,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:21,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:21,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:21,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832521328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832521329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,339 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,345 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,345 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,346 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,347 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,348 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,348 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,362 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c018cfe[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,367 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,373 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,373 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,378 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,379 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,379 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,380 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,384 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,386 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,386 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:21,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832521432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832521433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,486 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,491 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,492 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,492 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,492 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,495 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,495 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,495 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,495 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,497 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abfd0fe[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,502 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,502 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,505 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,505 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,505 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,505 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,505 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,508 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,510 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:21,510 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,511 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,511 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,511 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,511 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832521549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,611 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,614 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,619 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,619 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,619 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,619 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2662cb15[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,624 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,624 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,625 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,628 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,628 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,628 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,628 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,628 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,631 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,634 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:21,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,635 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,635 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832521635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832521636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,727 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d6b1b698b7964825a8de8999824ed718 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d6b1b698b7964825a8de8999824ed718 2023-04-07 01:54:21,735 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,739 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,741 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,743 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,743 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,743 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,743 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,745 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e035dd2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,754 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,758 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,758 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,768 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:21,768 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,868 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,871 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,871 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,871 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,871 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,875 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,875 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,875 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,875 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:21,875 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:21,877 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,877 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e454890[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:21,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:21,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,880 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:21,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:21,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:21,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:21,883 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:21,886 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:21,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:21,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:21,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:21,886 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:21,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:21,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:21,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:21,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:21,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:21,893 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:21,893 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:21,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:21,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:21,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832521939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832521939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,993 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:21,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:21,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:21,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:21,996 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:21,996 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,996 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:21,996 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:21,996 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,998 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:21,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:21,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:21,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:21,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:21,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:21,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:21,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:21,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:21,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:21,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:21,999 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:21,999 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:21,999 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:21,999 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a095b86[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:22,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:22,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,004 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:22,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:22,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:22,009 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:22,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:22,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:22,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:22,010 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:22,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,012 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:22,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,015 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:22,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,015 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,115 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:22,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:22,118 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:22,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:22,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:22,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,121 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,121 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:22,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:22,123 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,123 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:22,123 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:22,123 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0307b1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:22,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:22,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,129 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:22,129 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:22,131 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,133 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/87f280076c7a46e1b2af52a4d3e7caac 2023-04-07 01:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:22,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:22,134 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:22,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:22,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:22,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:22,134 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:22,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:22,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:22,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,160 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,160 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:22,160 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:22,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/968c4aef8f124bf58e8524f69b443598 2023-04-07 01:54:22,261 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:22,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:22,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:22,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:22,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:22,264 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:22,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:22,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:22,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:22,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,269 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:22,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8a24ee[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:22,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,272 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:22,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:22,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:22,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:22,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:22,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:22,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:22,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:22,281 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:22,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,286 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:22,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,286 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:22,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,386 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:22,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:22,389 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:22,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:22,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:22,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,391 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:22,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:22,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:22,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:22,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:22,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:22,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:22,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:22,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:22,401 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@affe267[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:22,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:22,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:22,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:22,415 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:22,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:22,419 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:22,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:22,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:22,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:22,420 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:22,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:22,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,439 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:22,439 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:22,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:22,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832522440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:22,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832522442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,539 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:22,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:22,543 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:22,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:22,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:22,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,547 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,548 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:22,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:22,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:22,550 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,550 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:22,550 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:22,550 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,550 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:22,550 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:22,550 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4a5dd5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:22,560 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:22,560 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:22,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:22,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:22,567 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:22,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:22,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:22,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:22,567 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:22,568 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,568 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,569 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,569 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:22,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:22,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:22,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:22,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:22,575 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:22,576 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:22,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:22,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/3c0233007c2448a083404f50b892d8a3 2023-04-07 01:54:22,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/87f280076c7a46e1b2af52a4d3e7caac as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac 2023-04-07 01:54:22,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac, entries=200, sequenceid=197, filesize=32.1 K 2023-04-07 01:54:22,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/968c4aef8f124bf58e8524f69b443598 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598 2023-04-07 01:54:22,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598, entries=150, sequenceid=197, filesize=11.8 K 2023-04-07 01:54:22,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/3c0233007c2448a083404f50b892d8a3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3 2023-04-07 01:54:22,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3, entries=150, sequenceid=197, filesize=11.8 K 2023-04-07 01:54:22,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 9bfd0d326e03d54dad8a7c3526a41584 in 1326ms, sequenceid=197, compaction requested=true 2023-04-07 01:54:22,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:22,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:22,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:22,615 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:22,615 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:22,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:22,616 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136526 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:22,616 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:22,616 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,617 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/56e6b508c8cb47edb9f86371f54b6880, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=133.3 K 2023-04-07 01:54:22,617 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60236 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:22,617 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:22,617 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 56e6b508c8cb47edb9f86371f54b6880, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:22,617 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,617 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0d0667ad81ec45cb9f8e21d6b97e3021, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=58.8 K 2023-04-07 01:54:22,617 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 96e8e3f3b1fe4829aa0223bd19a1e7b5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680832458118 2023-04-07 01:54:22,619 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9822d6543cdc4d9da50da471acdbaa3b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680832458755 2023-04-07 01:54:22,619 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0d0667ad81ec45cb9f8e21d6b97e3021, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:22,619 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 97cde93c85934d96ac564778773824e2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680832459409 2023-04-07 01:54:22,620 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 87f280076c7a46e1b2af52a4d3e7caac, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460965 2023-04-07 01:54:22,620 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 65ec605634c24342967d2d290184303f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680832458118 2023-04-07 01:54:22,620 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 47179463e513404f8a4cf0801b520a5e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680832458755 2023-04-07 01:54:22,621 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0b5481465e5642308082d21777322fa3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680832459409 2023-04-07 01:54:22,621 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 968c4aef8f124bf58e8524f69b443598, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460967 2023-04-07 01:54:22,642 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#151 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:22,643 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#150 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:22,676 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:22,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:22,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:22,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:22,683 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:22,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:22,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:22,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:22,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:22,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:22,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:22,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:22,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:22,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:22,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:22,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:22,687 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:22,688 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:22,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:22,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:22,688 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:22,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/ce57848b0e81421bbfdc903e153730f7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce57848b0e81421bbfdc903e153730f7 2023-04-07 01:54:22,703 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into ce57848b0e81421bbfdc903e153730f7(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:22,703 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:22,703 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=11, startTime=1680832462614; duration=0sec 2023-04-07 01:54:22,703 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:22,703 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:22,705 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60236 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:22,705 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:22,705 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:22,705 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30f699eece8c4de59d0eae48eada088f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=58.8 K 2023-04-07 01:54:22,706 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 30f699eece8c4de59d0eae48eada088f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680832456779 2023-04-07 01:54:22,706 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4a16b2692ca347a3911d1062c44ca8b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680832458118 2023-04-07 01:54:22,707 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0ee121acb51b4113879324dafb26b152, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680832458755 2023-04-07 01:54:22,707 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 13e8a8d1629943b89e63f6b8777b2b88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680832459409 2023-04-07 01:54:22,708 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3c0233007c2448a083404f50b892d8a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460967 2023-04-07 01:54:22,726 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#153 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:22,735 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d2317555eb4446b7adf2e112e3cc2037 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d2317555eb4446b7adf2e112e3cc2037 2023-04-07 01:54:22,754 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/e1a2923902584437989f8c5648b84baa 2023-04-07 01:54:23,064 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/564d6cd4be524d108c0508d2e0ca7861 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/564d6cd4be524d108c0508d2e0ca7861 2023-04-07 01:54:23,069 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into 564d6cd4be524d108c0508d2e0ca7861(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:23,069 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:23,069 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=11, startTime=1680832462615; duration=0sec 2023-04-07 01:54:23,069 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:23,144 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/6f7b4ea949094b3c8822915d82c25b89 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/6f7b4ea949094b3c8822915d82c25b89 2023-04-07 01:54:23,169 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 6f7b4ea949094b3c8822915d82c25b89(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:23,169 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:23,169 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=11, startTime=1680832462615; duration=0sec 2023-04-07 01:54:23,169 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:23,179 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/9d790e29ee3c4194990bf2981158fe7c 2023-04-07 01:54:23,193 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/7d132cf155974cc0bb2a0c8495af537e 2023-04-07 01:54:23,198 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/e1a2923902584437989f8c5648b84baa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa 2023-04-07 01:54:23,203 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa, entries=150, sequenceid=215, filesize=25.3 K 2023-04-07 01:54:23,204 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/9d790e29ee3c4194990bf2981158fe7c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c 2023-04-07 01:54:23,208 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c, entries=150, sequenceid=215, filesize=11.8 K 2023-04-07 01:54:23,209 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/7d132cf155974cc0bb2a0c8495af537e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e 2023-04-07 01:54:23,219 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e, entries=150, sequenceid=215, filesize=11.8 K 2023-04-07 01:54:23,220 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 9bfd0d326e03d54dad8a7c3526a41584 in 532ms, sequenceid=215, compaction requested=false 2023-04-07 01:54:23,220 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:23,220 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6876e1a2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,241 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,241 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,249 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:23,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,250 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,250 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,350 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:23,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:23,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:23,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:23,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:23,354 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:23,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:23,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:23,355 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,355 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,356 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:23,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:23,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:23,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:23,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f41516[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,363 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,363 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,367 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,370 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,370 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,374 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,377 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,377 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,477 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:23,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:23,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:23,481 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:23,481 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,481 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:23,481 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:23,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,484 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,485 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,485 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:23,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:23,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:23,486 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,486 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:23,487 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:23,487 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,488 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,488 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ed2b55[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,499 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,499 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:23,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,510 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,510 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:23,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:23,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:23,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832463589, max timestamp=1680832463595Num uniques 859; 2023-04-07 01:54:23,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:23,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832463589, max timestamp=1680832463595Num uniques 859; 2023-04-07 01:54:23,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:23,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832463589, max timestamp=1680832463595Num uniques 859; 2023-04-07 01:54:23,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:23,610 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:23,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:23,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:23,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:23,613 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:23,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:23,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:23,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:23,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:23,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:23,618 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,618 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:23,618 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:23,618 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832523616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832523617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a716c28[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832523621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,628 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,632 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,633 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,633 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,633 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,633 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,640 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,640 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832523723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832523722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832523725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,740 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:23,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:23,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:23,744 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:23,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:23,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:23,751 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,751 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,753 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:23,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:23,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:23,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:23,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:23,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,759 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d5f735[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,761 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,761 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,763 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,769 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,769 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,775 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:23,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,779 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,779 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832523837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,840 DEBUG [Thread-909] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:23,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832523853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,859 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:23,879 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:23,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:23,884 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:23,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:23,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:23,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,886 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:23,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:23,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:23,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:23,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:23,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:23,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:23,891 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:23,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326e043c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:23,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:23,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,895 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:23,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:23,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:23,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:23,900 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:23,900 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:23,900 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:23,900 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:23,900 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:23,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:23,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,905 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:23,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:23,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:23,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:23,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:23,908 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:23,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:23,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:23,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:23,908 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:23,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:23,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832523925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832523926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:23,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832523931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:23,975 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407fa3e77a2a0a14bd4904a9cfc9483b56f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fa3e77a2a0a14bd4904a9cfc9483b56f 2023-04-07 01:54:23,980 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/7c4b36cda0fe49fc884bb85df634ec7b 2023-04-07 01:54:24,008 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,015 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,018 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,026 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,028 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690226a7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,028 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,029 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,029 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,031 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,034 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,034 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,048 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,048 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,149 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,152 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,153 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,153 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,155 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,156 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,156 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,157 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,157 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68131783[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,160 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,165 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,168 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,168 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,170 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,170 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,170 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,176 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,176 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832524228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832524230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832524237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,276 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,279 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,285 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,285 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128529eb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,288 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,288 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,291 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,293 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,293 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,296 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,301 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,301 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,301 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/c03ed4f6d37e4f0e92516d5d67a79c7d 2023-04-07 01:54:24,401 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,408 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,412 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,412 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,412 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,412 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50089063[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,419 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,422 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,422 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,425 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,436 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,436 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,437 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,537 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,540 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,540 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2847b3db[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,553 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,554 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,554 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,554 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,554 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:24,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,559 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,560 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,660 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,663 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,667 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,667 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,667 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,667 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5818165f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,676 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,680 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,680 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,682 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,687 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,687 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832524733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832524735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832524745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,787 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,790 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,792 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:24,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:24,796 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6369b6c6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:24,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:24,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,799 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:24,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:24,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:24,804 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:24,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:24,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:24,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:24,805 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:24,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:24,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:24,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:24,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,814 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:24,814 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:24,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:24,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/30d355b663304b8d88c759607bd291e3 2023-04-07 01:54:24,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/7c4b36cda0fe49fc884bb85df634ec7b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b 2023-04-07 01:54:24,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b, entries=200, sequenceid=231, filesize=32.1 K 2023-04-07 01:54:24,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/c03ed4f6d37e4f0e92516d5d67a79c7d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d 2023-04-07 01:54:24,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d, entries=150, sequenceid=231, filesize=11.8 K 2023-04-07 01:54:24,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/30d355b663304b8d88c759607bd291e3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3 2023-04-07 01:54:24,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3, entries=150, sequenceid=231, filesize=11.8 K 2023-04-07 01:54:24,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9bfd0d326e03d54dad8a7c3526a41584 in 1292ms, sequenceid=231, compaction requested=true 2023-04-07 01:54:24,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:24,846 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:24,846 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:24,847 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:24,847 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:24,847 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,847 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce57848b0e81421bbfdc903e153730f7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=83.1 K 2023-04-07 01:54:24,848 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:24,848 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:24,848 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,848 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/564d6cd4be524d108c0508d2e0ca7861, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=35.6 K 2023-04-07 01:54:24,848 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ce57848b0e81421bbfdc903e153730f7, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460967 2023-04-07 01:54:24,849 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 564d6cd4be524d108c0508d2e0ca7861, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460967 2023-04-07 01:54:24,849 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e1a2923902584437989f8c5648b84baa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680832461300 2023-04-07 01:54:24,850 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9d790e29ee3c4194990bf2981158fe7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680832461300 2023-04-07 01:54:24,850 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7c4b36cda0fe49fc884bb85df634ec7b, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:24,850 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c03ed4f6d37e4f0e92516d5d67a79c7d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:24,862 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:24,865 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#159 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:24,883 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/02609460a8f1476ca2e04996f60d369f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/02609460a8f1476ca2e04996f60d369f 2023-04-07 01:54:24,889 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into 02609460a8f1476ca2e04996f60d369f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:24,889 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:24,889 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=13, startTime=1680832464846; duration=0sec 2023-04-07 01:54:24,889 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:24,889 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:24,890 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:24,890 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:24,890 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,890 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/6f7b4ea949094b3c8822915d82c25b89, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=35.6 K 2023-04-07 01:54:24,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6f7b4ea949094b3c8822915d82c25b89, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680832460967 2023-04-07 01:54:24,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7d132cf155974cc0bb2a0c8495af537e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680832461300 2023-04-07 01:54:24,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 30d355b663304b8d88c759607bd291e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:24,897 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:24,914 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:24,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:24,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:24,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:24,918 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:24,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:24,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:24,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:24,920 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:24,920 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:24,920 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:24,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:24,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:24,922 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:24,922 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:24,922 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-07 01:54:24,922 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:24,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:24,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:24,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:24,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:24,923 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:25,300 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/38248deb3a2449eab9f1cc26a0222666 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/38248deb3a2449eab9f1cc26a0222666 2023-04-07 01:54:25,305 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into 38248deb3a2449eab9f1cc26a0222666(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:25,305 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:25,305 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=13, startTime=1680832464846; duration=0sec 2023-04-07 01:54:25,306 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:25,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/374bf860d3d347eab7d307bb266dbb49 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/374bf860d3d347eab7d307bb266dbb49 2023-04-07 01:54:25,317 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 374bf860d3d347eab7d307bb266dbb49(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:25,317 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:25,317 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=13, startTime=1680832464846; duration=0sec 2023-04-07 01:54:25,317 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:25,341 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304075f8b0851fb144dbfa03c5fcc0f097f96 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075f8b0851fb144dbfa03c5fcc0f097f96 2023-04-07 01:54:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:25,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:25,748 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=267, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/60a499c91c434575880892b14cb3fd5e 2023-04-07 01:54:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:25,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832525745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:25,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:25,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48632 deadline: 1680832525749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:25,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832525754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:25,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832525850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:25,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832525856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832526056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832526060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,161 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/e8fa39f0b8d44192bdc96c9ee9acbc98 2023-04-07 01:54:26,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832526360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48636 deadline: 1680832526362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,572 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/b14e1717c2c1460a9c652509916f1ceb 2023-04-07 01:54:26,579 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/60a499c91c434575880892b14cb3fd5e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e 2023-04-07 01:54:26,590 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e, entries=150, sequenceid=267, filesize=25.4 K 2023-04-07 01:54:26,591 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/e8fa39f0b8d44192bdc96c9ee9acbc98 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98 2023-04-07 01:54:26,596 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98, entries=150, sequenceid=267, filesize=11.9 K 2023-04-07 01:54:26,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/b14e1717c2c1460a9c652509916f1ceb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb 2023-04-07 01:54:26,603 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb, entries=150, sequenceid=267, filesize=11.9 K 2023-04-07 01:54:26,604 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9bfd0d326e03d54dad8a7c3526a41584 in 1682ms, sequenceid=267, compaction requested=false 2023-04-07 01:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:26,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f6b940[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:26,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:26,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:26,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:26,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:26,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:26,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:26,622 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:26,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:26,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:26,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:26,622 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:26,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,631 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:26,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,635 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:26,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:26,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:26,636 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:26,636 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:26,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:26,736 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:26,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:26,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:26,740 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:26,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:26,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:26,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:26,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:26,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,742 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:26,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:26,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:26,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:26,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:26,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:26,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:26,745 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:26,745 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:26,745 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:26,745 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:26,746 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:26,746 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:26,746 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:26,746 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:26,746 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:26,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:26,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:26,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832466877, max timestamp=1680832466892Num uniques 859; 2023-04-07 01:54:26,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832466877, max timestamp=1680832466892Num uniques 859; 2023-04-07 01:54:26,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:26,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:26,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832466877, max timestamp=1680832466892Num uniques 859; 2023-04-07 01:54:26,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:54:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832526911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:26,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:26,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832526912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832527015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832527015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,169 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407b494f1fa4b184ef39d3e9bfe351950e4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b494f1fa4b184ef39d3e9bfe351950e4 2023-04-07 01:54:27,174 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/ce59466fbe2e476d9c897d575bc36a17 2023-04-07 01:54:27,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832527219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832527219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832527522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832527523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,589 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/73503a33b43949aaa8e7cecf8f4f2f8d 2023-04-07 01:54:27,604 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/5a04a21d47e34775820adae4a7b1c697 2023-04-07 01:54:27,613 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/ce59466fbe2e476d9c897d575bc36a17 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17 2023-04-07 01:54:27,618 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17, entries=150, sequenceid=280, filesize=25.4 K 2023-04-07 01:54:27,619 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/73503a33b43949aaa8e7cecf8f4f2f8d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d 2023-04-07 01:54:27,625 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d, entries=150, sequenceid=280, filesize=11.9 K 2023-04-07 01:54:27,626 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/5a04a21d47e34775820adae4a7b1c697 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697 2023-04-07 01:54:27,630 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697, entries=150, sequenceid=280, filesize=11.9 K 2023-04-07 01:54:27,632 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9bfd0d326e03d54dad8a7c3526a41584 in 887ms, sequenceid=280, compaction requested=true 2023-04-07 01:54:27,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:27,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:27,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:27,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623a050e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:27,635 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:27,635 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,638 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:27,640 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:27,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:27,640 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:27,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:27,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,645 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,648 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:27,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:27,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:27,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:27,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:27,649 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:27,650 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:27,750 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:27,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:27,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:27,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:27,753 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:27,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:27,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:27,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:27,755 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,755 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,758 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:27,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:27,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:27,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:27,760 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:27,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:27,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:27,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:27,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:27,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:27,778 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040719688416b006421e9b57ee19fe695634 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719688416b006421e9b57ee19fe695634 2023-04-07 01:54:27,785 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/f1d85e148af347d3b5a96e7e35046e56 2023-04-07 01:54:27,797 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/59c7b9aa642a4bbcb43caa5747568975 2023-04-07 01:54:27,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:27,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48662 deadline: 1680832527795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,820 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/c59f5d1127b84a6288b66ffa1c20c1db 2023-04-07 01:54:27,826 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/f1d85e148af347d3b5a96e7e35046e56 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56 2023-04-07 01:54:27,833 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56, entries=150, sequenceid=314, filesize=25.4 K 2023-04-07 01:54:27,834 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/59c7b9aa642a4bbcb43caa5747568975 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975 2023-04-07 01:54:27,840 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975, entries=150, sequenceid=314, filesize=11.9 K 2023-04-07 01:54:27,841 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/c59f5d1127b84a6288b66ffa1c20c1db as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db 2023-04-07 01:54:27,849 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db, entries=150, sequenceid=314, filesize=11.9 K 2023-04-07 01:54:27,850 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 9bfd0d326e03d54dad8a7c3526a41584 in 90ms, sequenceid=314, compaction requested=true 2023-04-07 01:54:27,850 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:27,851 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:27,852 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:27,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a884df8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:27,855 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:27,855 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:27,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:27,858 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,858 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:27,859 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,859 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:27,861 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:27,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:27,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:27,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:27,862 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:27,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,869 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,870 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:27,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:27,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:27,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:27,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:27,874 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:27,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,874 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:27,874 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,874 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:27,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:54:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:27,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:27,974 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:27,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:27,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:27,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:27,981 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:27,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:27,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:27,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:27,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,983 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:27,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:27,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:27,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:27,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:27,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:27,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:27,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:27,989 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:27,989 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:27,989 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:27,989 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:27,989 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:27,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:27,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54791be9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:27,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:27,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,998 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:27,999 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:27,999 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:27,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:27,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,000 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,002 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,003 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,005 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,005 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,005 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,011 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,014 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,014 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832528023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832528028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832528032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,114 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,118 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,120 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,122 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,122 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,122 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,122 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,123 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1687949f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,127 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,129 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,131 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,131 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,131 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,131 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,131 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832528135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,138 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:28,138 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,138 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,239 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,252 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,252 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,253 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,253 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,257 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,257 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,257 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,257 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,259 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2427e134[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,262 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,262 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,263 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,264 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,264 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,264 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,276 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,276 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,285 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,285 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,335 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304070f7930c5f341494ea08a66b6472cc150 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f7930c5f341494ea08a66b6472cc150 2023-04-07 01:54:28,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832528339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,385 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,393 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,394 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,395 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4b260b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,420 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,420 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,428 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,428 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:28,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,528 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,532 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,537 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,537 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,538 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,538 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,540 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e4c37[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,548 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,555 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,556 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,560 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,563 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,563 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,564 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:28,564 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:28,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832528643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,665 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,668 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69206b7a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,689 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,689 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,692 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,695 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,696 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,703 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,706 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,710 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,711 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,711 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,711 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,740 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/769e248df9c44d0eb7fe58d463a456b2 2023-04-07 01:54:28,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/9412a22ab6754d7abc2ee51ffa46ed8e 2023-04-07 01:54:28,811 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,815 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,818 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,818 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,818 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,819 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7abe02c7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,822 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,822 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,823 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,830 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,830 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,832 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,837 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,837 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,937 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:28,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:28,940 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:28,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:28,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:28,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:28,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:28,944 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,944 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:28,944 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:28,944 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:28,947 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b608e5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:28,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:28,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:28,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:28,951 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,951 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:28,954 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:28,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:28,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:28,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:28,954 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:28,956 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,956 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,956 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:28,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:28,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:28,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:28,961 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:28,961 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:28,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:28,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:28,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:28,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:29,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832529040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:29,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832529046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,061 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,064 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:29,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6ae210[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,075 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,081 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,081 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,088 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,088 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:29,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832529148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/df92165f13844f4cbbc6c3c7dcd5e19a 2023-04-07 01:54:29,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/769e248df9c44d0eb7fe58d463a456b2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2 2023-04-07 01:54:29,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2, entries=150, sequenceid=326, filesize=25.4 K 2023-04-07 01:54:29,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/9412a22ab6754d7abc2ee51ffa46ed8e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e 2023-04-07 01:54:29,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e, entries=100, sequenceid=326, filesize=9.5 K 2023-04-07 01:54:29,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/df92165f13844f4cbbc6c3c7dcd5e19a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a 2023-04-07 01:54:29,188 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a, entries=100, sequenceid=326, filesize=9.5 K 2023-04-07 01:54:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9bfd0d326e03d54dad8a7c3526a41584 in 1278ms, sequenceid=326, compaction requested=true 2023-04-07 01:54:29,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:29,190 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:29,190 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:29,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:29,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:29,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,192 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130504 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:29,192 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:29,192 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,192 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/38248deb3a2449eab9f1cc26a0222666, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=127.4 K 2023-04-07 01:54:29,193 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58764 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:29,193 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:29,193 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,193 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/02609460a8f1476ca2e04996f60d369f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=57.4 K 2023-04-07 01:54:29,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,193 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,193 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 02609460a8f1476ca2e04996f60d369f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:29,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,194 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 38248deb3a2449eab9f1cc26a0222666, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:29,194 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e8fa39f0b8d44192bdc96c9ee9acbc98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680832463605 2023-04-07 01:54:29,194 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 60a499c91c434575880892b14cb3fd5e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680832463605 2023-04-07 01:54:29,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 73503a33b43949aaa8e7cecf8f4f2f8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680832465743 2023-04-07 01:54:29,195 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ce59466fbe2e476d9c897d575bc36a17, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680832465743 2023-04-07 01:54:29,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 59c7b9aa642a4bbcb43caa5747568975, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680832466904 2023-04-07 01:54:29,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,195 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f1d85e148af347d3b5a96e7e35046e56, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680832466904 2023-04-07 01:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9412a22ab6754d7abc2ee51ffa46ed8e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832467793 2023-04-07 01:54:29,196 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 769e248df9c44d0eb7fe58d463a456b2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832467788 2023-04-07 01:54:29,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,205 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,205 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,205 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:54:29,205 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:29,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:29,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:29,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:29,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:29,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:29,207 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#174 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:29,227 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#175 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:29,238 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/3fa14ec73fbb445bb2ac41383b90eda6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3fa14ec73fbb445bb2ac41383b90eda6 2023-04-07 01:54:29,250 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into 3fa14ec73fbb445bb2ac41383b90eda6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:29,250 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,250 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=11, startTime=1680832469190; duration=0sec 2023-04-07 01:54:29,251 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:29,251 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:29,253 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58764 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:29,253 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:29,253 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,254 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/374bf860d3d347eab7d307bb266dbb49, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=57.4 K 2023-04-07 01:54:29,254 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 374bf860d3d347eab7d307bb266dbb49, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832463517 2023-04-07 01:54:29,254 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b14e1717c2c1460a9c652509916f1ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680832463605 2023-04-07 01:54:29,255 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5a04a21d47e34775820adae4a7b1c697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680832465743 2023-04-07 01:54:29,255 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c59f5d1127b84a6288b66ffa1c20c1db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680832466904 2023-04-07 01:54:29,256 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting df92165f13844f4cbbc6c3c7dcd5e19a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832467793 2023-04-07 01:54:29,267 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407c6adf72af95d4775a135de78e8e31f57 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c6adf72af95d4775a135de78e8e31f57 2023-04-07 01:54:29,267 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#177 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:29,278 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/16f203a3a7c34470a14391187fab1a06 2023-04-07 01:54:29,296 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/512101e32b5d454f9ef39b091ab596bc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/512101e32b5d454f9ef39b091ab596bc 2023-04-07 01:54:29,298 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/0a3f416d798f49959a644500f762daf3 2023-04-07 01:54:29,304 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 512101e32b5d454f9ef39b091ab596bc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:29,304 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,304 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=11, startTime=1680832469190; duration=0sec 2023-04-07 01:54:29,304 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:29,312 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/52da7dd870254096ac84a93b723cc7b6 2023-04-07 01:54:29,317 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/16f203a3a7c34470a14391187fab1a06 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06 2023-04-07 01:54:29,321 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06, entries=150, sequenceid=346, filesize=25.4 K 2023-04-07 01:54:29,323 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/0a3f416d798f49959a644500f762daf3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3 2023-04-07 01:54:29,328 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3, entries=150, sequenceid=346, filesize=11.9 K 2023-04-07 01:54:29,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/52da7dd870254096ac84a93b723cc7b6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6 2023-04-07 01:54:29,335 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6, entries=150, sequenceid=346, filesize=11.9 K 2023-04-07 01:54:29,335 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 9bfd0d326e03d54dad8a7c3526a41584 in 130ms, sequenceid=346, compaction requested=false 2023-04-07 01:54:29,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fb3b9c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,344 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,347 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,347 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,350 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:29,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,360 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,360 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,460 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,465 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,467 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:29,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad9ad7e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,485 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,485 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,485 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,485 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,485 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,495 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,495 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:29,595 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,599 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,600 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,600 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,606 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139169e0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,610 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,613 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,613 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,623 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,624 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,724 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,731 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,732 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:29,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,735 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,735 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,735 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,735 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,737 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7673e0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,742 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,742 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,745 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,748 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,748 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,762 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,762 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,863 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:29,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:29,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:29,867 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:29,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:29,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,870 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,870 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,871 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,871 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:29,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:29,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:29,888 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,888 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:29,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:29,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:29,891 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:29,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a89e69[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:29,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:29,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,896 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:29,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:29,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:29,907 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:29,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:29,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:29,907 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:29,907 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:29,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:29,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:29,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:29,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:29,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:29,915 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:29,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:29,915 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:29,915 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:29,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:30,016 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:30,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:30,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:30,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:30,020 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:30,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:30,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:30,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:30,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,021 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:30,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:30,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:30,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:30,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:30,023 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:30,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:30,024 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:30,025 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:30,025 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:30,026 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:30,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:30,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107d7fbf[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:30,029 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:30,029 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,032 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:30,032 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,032 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:30,033 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:30,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:30,036 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:30,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:30,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:30,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:30,037 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:30,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:30,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:30,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:30,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:30,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:30,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:30,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:30,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:30,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,043 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:30,043 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:30,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:30,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:30,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:30,110 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/765d0b90241b47f2a99968be7187c8ff as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/765d0b90241b47f2a99968be7187c8ff 2023-04-07 01:54:30,114 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into 765d0b90241b47f2a99968be7187c8ff(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:30,115 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:30,115 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=11, startTime=1680832469190; duration=0sec 2023-04-07 01:54:30,115 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:30,143 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:30,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:30,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:30,148 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:30,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:30,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:30,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:30,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:30,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:30,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:30,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:30,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:30,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:30,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:30,172 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:30,173 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:30,202 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304071dbfc136262f4022abd1ed70ddcd0dff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071dbfc136262f4022abd1ed70ddcd0dff 2023-04-07 01:54:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:30,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832470291, max timestamp=1680832470301Num uniques 859; 2023-04-07 01:54:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832470291, max timestamp=1680832470301Num uniques 859; 2023-04-07 01:54:30,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:30,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832470291, max timestamp=1680832470301Num uniques 859; 2023-04-07 01:54:30,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:30,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832470318, max timestamp=1680832470349Num uniques 1018; 2023-04-07 01:54:30,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:30,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832470318, max timestamp=1680832470349Num uniques 1018; 2023-04-07 01:54:30,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:30,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832470318, max timestamp=1680832470349Num uniques 1018; 2023-04-07 01:54:30,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832470374, max timestamp=1680832470406Num uniques 1077; 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832470374, max timestamp=1680832470406Num uniques 1077; 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832470374, max timestamp=1680832470406Num uniques 1077; 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-07 01:54:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-07 01:54:30,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832470443, max timestamp=1680832470450Num uniques 1086; 2023-04-07 01:54:30,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:30,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832470443, max timestamp=1680832470450Num uniques 1086; 2023-04-07 01:54:30,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:30,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832470443, max timestamp=1680832470450Num uniques 1086; 2023-04-07 01:54:30,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:30,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832470490, max timestamp=1680832470500Num uniques 1045; 2023-04-07 01:54:30,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-07 01:54:30,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832470490, max timestamp=1680832470500Num uniques 1045; 2023-04-07 01:54:30,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-07 01:54:30,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832470490, max timestamp=1680832470500Num uniques 1045; 2023-04-07 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-07 01:54:30,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832470513, max timestamp=1680832470555Num uniques 1104; 2023-04-07 01:54:30,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832470513, max timestamp=1680832470555Num uniques 1104; 2023-04-07 01:54:30,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:54:30,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:54:30,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832470513, max timestamp=1680832470555Num uniques 1104; 2023-04-07 01:54:30,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:54:30,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832470591, max timestamp=1680832470601Num uniques 1113; 2023-04-07 01:54:30,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-07 01:54:30,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832470591, max timestamp=1680832470601Num uniques 1113; 2023-04-07 01:54:30,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-07 01:54:30,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832470591, max timestamp=1680832470601Num uniques 1113; 2023-04-07 01:54:30,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-07 01:54:30,635 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/3c8ab13119914f74bd66365fd9fe80ee 2023-04-07 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832470640, max timestamp=1680832470651Num uniques 1072; 2023-04-07 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832470640, max timestamp=1680832470651Num uniques 1072; 2023-04-07 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832470640, max timestamp=1680832470651Num uniques 1072; 2023-04-07 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-07 01:54:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-07 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-07 01:54:30,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832470683, max timestamp=1680832470712Num uniques 1031; 2023-04-07 01:54:30,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-07 01:54:30,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832470683, max timestamp=1680832470712Num uniques 1031; 2023-04-07 01:54:30,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-07 01:54:30,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832470701, max timestamp=1680832470712Num uniques 1031; 2023-04-07 01:54:30,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:54:30,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832470752, max timestamp=1680832470766Num uniques 1090; 2023-04-07 01:54:30,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-07 01:54:30,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832470752, max timestamp=1680832470766Num uniques 1090; 2023-04-07 01:54:30,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832470752, max timestamp=1680832470766Num uniques 1040; 2023-04-07 01:54:30,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-07 01:54:30,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-07 01:54:30,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832470794, max timestamp=1680832470802Num uniques 1099; 2023-04-07 01:54:30,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:54:30,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832470794, max timestamp=1680832470802Num uniques 1099; 2023-04-07 01:54:30,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:54:30,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832470794, max timestamp=1680832470802Num uniques 1099; 2023-04-07 01:54:30,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:54:30,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832470815, max timestamp=1680832470861Num uniques 1058; 2023-04-07 01:54:30,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:54:30,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832470815, max timestamp=1680832470861Num uniques 1058; 2023-04-07 01:54:30,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:54:30,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832470815, max timestamp=1680832470861Num uniques 1058; 2023-04-07 01:54:30,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:54:30,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832470894, max timestamp=1680832470917Num uniques 1067; 2023-04-07 01:54:30,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:54:30,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832470894, max timestamp=1680832470917Num uniques 1067; 2023-04-07 01:54:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:54:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832470894, max timestamp=1680832470917Num uniques 1067; 2023-04-07 01:54:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832470945, max timestamp=1680832470957Num uniques 1076; 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832470945, max timestamp=1680832470957Num uniques 1076; 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832470945, max timestamp=1680832470957Num uniques 1076; 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:54:30,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832470982, max timestamp=1680832471001Num uniques 1035; 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832470982, max timestamp=1680832471001Num uniques 1035; 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832470982, max timestamp=1680832471001Num uniques 1035; 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-07 01:54:31,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-07 01:54:31,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680832471019, max timestamp=1680832471041Num uniques 1094; 2023-04-07 01:54:31,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-07 01:54:31,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680832471019, max timestamp=1680832471041Num uniques 1094; 2023-04-07 01:54:31,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680832471019, max timestamp=1680832471041Num uniques 1094; 2023-04-07 01:54:31,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-07 01:54:31,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-07 01:54:31,049 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/057e5ad240f5444f84442ccb7eab9c9a 2023-04-07 01:54:31,065 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/f75bf3ebe417488597109c325511f81c 2023-04-07 01:54:31,070 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/3c8ab13119914f74bd66365fd9fe80ee as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee 2023-04-07 01:54:31,076 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee, entries=50, sequenceid=358, filesize=11.7 K 2023-04-07 01:54:31,077 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/057e5ad240f5444f84442ccb7eab9c9a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a 2023-04-07 01:54:31,080 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a, entries=50, sequenceid=358, filesize=7.1 K 2023-04-07 01:54:31,081 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/f75bf3ebe417488597109c325511f81c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c 2023-04-07 01:54:31,085 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c, entries=50, sequenceid=358, filesize=7.1 K 2023-04-07 01:54:31,085 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=134.18 KB/137400 for 9bfd0d326e03d54dad8a7c3526a41584 in 913ms, sequenceid=358, compaction requested=true 2023-04-07 01:54:31,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:31,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:31,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=140.89 KB heapSize=401.63 KB 2023-04-07 01:54:31,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832471069, max timestamp=1680832471087Num uniques 1053; 2023-04-07 01:54:31,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-07 01:54:31,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832471069, max timestamp=1680832471087Num uniques 1053; 2023-04-07 01:54:31,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-07 01:54:31,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:31,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:31,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:31,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832471069, max timestamp=1680832471087Num uniques 1053; 2023-04-07 01:54:31,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-07 01:54:31,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83bca39[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:31,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:31,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:31,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:31,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:31,102 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:31,104 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:31,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:31,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:31,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:31,104 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:31,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,112 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,115 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:31,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,115 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,116 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,116 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,116 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,116 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:31,116 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:31,125 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407babbe4d0b8234e5fb04aa8161fab5e84 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407babbe4d0b8234e5fb04aa8161fab5e84 2023-04-07 01:54:31,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832471117, max timestamp=1680832471123Num uniques 859; 2023-04-07 01:54:31,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:31,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832471117, max timestamp=1680832471123Num uniques 859; 2023-04-07 01:54:31,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:31,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832471117, max timestamp=1680832471123Num uniques 859; 2023-04-07 01:54:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:31,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680832471130, max timestamp=1680832471149Num uniques 1018; 2023-04-07 01:54:31,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-07 01:54:31,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680832471130, max timestamp=1680832471149Num uniques 1018; 2023-04-07 01:54:31,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680832471130, max timestamp=1680832471149Num uniques 1018; 2023-04-07 01:54:31,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-07 01:54:31,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-07 01:54:31,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680832471166, max timestamp=1680832471175Num uniques 1177; 2023-04-07 01:54:31,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-07 01:54:31,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680832471166, max timestamp=1680832471175Num uniques 1177; 2023-04-07 01:54:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680832471166, max timestamp=1680832471175Num uniques 1177; 2023-04-07 01:54:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-07 01:54:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-07 01:54:31,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832531202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680832471189, max timestamp=1680832471203Num uniques 1136; 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680832471189, max timestamp=1680832471203Num uniques 1136; 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680832471189, max timestamp=1680832471203Num uniques 1136; 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-07 01:54:31,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-07 01:54:31,216 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:31,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:31,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:31,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:31,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:31,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:31,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:31,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:31,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:31,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:31,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:31,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:31,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ef41ad[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:31,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:31,238 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,238 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:31,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:31,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832531236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:31,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832531237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:31,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:31,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:31,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:31,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:31,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:31,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:31,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,248 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:31,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,248 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:31,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,248 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:31,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832531305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832531340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832531341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,348 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:31,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:31,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:31,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:31,351 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:31,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:31,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:31,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,354 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,355 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,355 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:31,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:31,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:31,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:31,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:31,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:31,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:31,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@687daaf[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:31,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:31,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:31,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:31,362 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,362 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:31,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:31,363 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:31,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:31,365 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:31,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:31,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:31,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:31,366 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:31,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,369 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:31,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:31,372 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:31,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,373 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:31,373 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:31,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,473 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:31,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:31,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:31,477 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:31,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,478 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:31,478 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:31,478 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,478 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:31,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:31,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:31,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:31,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:31,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:31,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:31,484 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:31,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:31,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfccd47[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:31,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:31,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:31,494 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:31,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:31,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:31,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:31,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:31,498 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:31,498 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:31,498 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:31,498 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:31,498 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:31,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:31,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,500 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:31,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:31,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:31,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:31,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:31,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:31,504 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:31,504 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:31,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:31,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832531519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,532 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=654, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/fae78cc2a46e4a509a8a4175764fd441 2023-04-07 01:54:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832531543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832531545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/dd7773765cd94c9b82cd4d2d07c65a92 2023-04-07 01:54:31,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/2412e818a01f461c8d4f01c2edf68793 2023-04-07 01:54:31,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/fae78cc2a46e4a509a8a4175764fd441 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441 2023-04-07 01:54:31,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441, entries=250, sequenceid=654, filesize=39.2 K 2023-04-07 01:54:31,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/dd7773765cd94c9b82cd4d2d07c65a92 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92 2023-04-07 01:54:31,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92, entries=150, sequenceid=654, filesize=11.9 K 2023-04-07 01:54:31,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/2412e818a01f461c8d4f01c2edf68793 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793 2023-04-07 01:54:31,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793, entries=150, sequenceid=654, filesize=11.9 K 2023-04-07 01:54:31,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~97.22 KB/99552, currentSize=154.31 KB/158010 for 9bfd0d326e03d54dad8a7c3526a41584 in 517ms, sequenceid=654, compaction requested=true 2023-04-07 01:54:31,604 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:31,605 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:31,605 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:31,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:31,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:31,606 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:31,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:31,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:31,607 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104826 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:31,607 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:31,607 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,607 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/765d0b90241b47f2a99968be7187c8ff, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=102.4 K 2023-04-07 01:54:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:31,607 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 765d0b90241b47f2a99968be7187c8ff, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832466910 2023-04-07 01:54:31,608 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 16f203a3a7c34470a14391187fab1a06, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680832468007 2023-04-07 01:54:31,608 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3c8ab13119914f74bd66365fd9fe80ee, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680832470170 2023-04-07 01:54:31,608 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:31,609 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:31,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:31,609 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,609 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3fa14ec73fbb445bb2ac41383b90eda6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=43.5 K 2023-04-07 01:54:31,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:31,609 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:31,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:31,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:31,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,611 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting fae78cc2a46e4a509a8a4175764fd441, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:31,611 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3fa14ec73fbb445bb2ac41383b90eda6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832466910 2023-04-07 01:54:31,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0a3f416d798f49959a644500f762daf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680832468007 2023-04-07 01:54:31,613 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 057e5ad240f5444f84442ccb7eab9c9a, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680832470170 2023-04-07 01:54:31,613 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dd7773765cd94c9b82cd4d2d07c65a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:31,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,613 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:31,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:31,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:31,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:31,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:31,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:31,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:31,616 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,616 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:31,616 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=154.31 KB heapSize=428.16 KB 2023-04-07 01:54:31,616 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:31,616 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,617 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:31,617 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,617 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:31,617 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:31,630 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:31,637 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#186 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:31,661 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e4184fd2e9914b3d99444e4690e85cb7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e4184fd2e9914b3d99444e4690e85cb7 2023-04-07 01:54:31,675 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/b1f560c88b204ffbac34c4b6b9f87acf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b1f560c88b204ffbac34c4b6b9f87acf 2023-04-07 01:54:31,680 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=742, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/5b7f1735ce194e8eaa05cc71c07e362a 2023-04-07 01:54:31,689 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into b1f560c88b204ffbac34c4b6b9f87acf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:31,689 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:31,689 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=12, startTime=1680832471606; duration=0sec 2023-04-07 01:54:31,689 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:31,690 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:31,692 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:31,692 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:31,692 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:31,692 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/512101e32b5d454f9ef39b091ab596bc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=43.5 K 2023-04-07 01:54:31,692 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 512101e32b5d454f9ef39b091ab596bc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680832466910 2023-04-07 01:54:31,693 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 52da7dd870254096ac84a93b723cc7b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680832468007 2023-04-07 01:54:31,693 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f75bf3ebe417488597109c325511f81c, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680832470170 2023-04-07 01:54:31,693 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2412e818a01f461c8d4f01c2edf68793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:31,705 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:31,719 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/74bdf2f9e51a44ac816411197323caa2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/74bdf2f9e51a44ac816411197323caa2 2023-04-07 01:54:31,724 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 74bdf2f9e51a44ac816411197323caa2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:31,724 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:31,724 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=12, startTime=1680832471606; duration=0sec 2023-04-07 01:54:31,724 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:31,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:31,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:31,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832531833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832531846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832531848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:31,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:31,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832531935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,114 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/d8e8184ca5f94ce48c2f36fd77f65d67 2023-04-07 01:54:32,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:32,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832532138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832532350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:32,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832532355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:32,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832532441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,534 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/1b32df062404498fa56fa0dfc9d105c1 2023-04-07 01:54:32,538 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/5b7f1735ce194e8eaa05cc71c07e362a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a 2023-04-07 01:54:32,542 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a, entries=150, sequenceid=742, filesize=25.4 K 2023-04-07 01:54:32,542 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/d8e8184ca5f94ce48c2f36fd77f65d67 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67 2023-04-07 01:54:32,546 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67, entries=150, sequenceid=742, filesize=11.9 K 2023-04-07 01:54:32,547 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/1b32df062404498fa56fa0dfc9d105c1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1 2023-04-07 01:54:32,550 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1, entries=150, sequenceid=742, filesize=11.9 K 2023-04-07 01:54:32,551 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.11 KB/438384, currentSize=33.54 KB/34350 for 9bfd0d326e03d54dad8a7c3526a41584 in 935ms, sequenceid=742, compaction requested=false 2023-04-07 01:54:32,551 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:32,551 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:32,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:32,554 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:32,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc0aac3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:32,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:32,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:32,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:32,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:32,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:32,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:32,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:32,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:32,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:32,567 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:32,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:32,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:32,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:32,567 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:32,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:32,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:32,571 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:32,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:32,574 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:32,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:32,574 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:32,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:32,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:32,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:32,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:32,575 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:32,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:32,575 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:32,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,675 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:32,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:32,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:32,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:32,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:32,678 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:32,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:32,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:32,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:32,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:32,680 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:32,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:32,681 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:32,681 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:32,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:32,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:32,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:32,682 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:32,682 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:32,682 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:32,683 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:32,900 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/e9f36690d9f54c7586212784fd70a2d7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e9f36690d9f54c7586212784fd70a2d7 2023-04-07 01:54:32,904 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into e9f36690d9f54c7586212784fd70a2d7(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-07 01:54:32,905 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:32,905 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=12, startTime=1680832471605; duration=1sec 2023-04-07 01:54:32,905 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:32,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:32,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832472960, max timestamp=1680832472978Num uniques 859; 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832472960, max timestamp=1680832472978Num uniques 859; 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832472960, max timestamp=1680832472978Num uniques 859; 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:32,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832472985, max timestamp=1680832473018Num uniques 1018; 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832472985, max timestamp=1680832473018Num uniques 1018; 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832472985, max timestamp=1680832473018Num uniques 1018; 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:33,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:33,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832473040, max timestamp=1680832473075Num uniques 1027; 2023-04-07 01:54:33,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 01:54:33,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832473040, max timestamp=1680832473075Num uniques 1027; 2023-04-07 01:54:33,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 01:54:33,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680832473040, max timestamp=1680832473075Num uniques 1027; 2023-04-07 01:54:33,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 01:54:33,101 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407c7eec748d2784d2386f9e0d56f0be21b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c7eec748d2784d2386f9e0d56f0be21b 2023-04-07 01:54:33,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832473102, max timestamp=1680832473132Num uniques 1086; 2023-04-07 01:54:33,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:33,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832473102, max timestamp=1680832473132Num uniques 1086; 2023-04-07 01:54:33,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:33,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832473102, max timestamp=1680832473132Num uniques 1086; 2023-04-07 01:54:33,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 01:54:33,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832473165, max timestamp=1680832473173Num uniques 1045; 2023-04-07 01:54:33,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:54:33,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832473165, max timestamp=1680832473173Num uniques 1045; 2023-04-07 01:54:33,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:54:33,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832473165, max timestamp=1680832473173Num uniques 1045; 2023-04-07 01:54:33,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:54:33,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680832473202, max timestamp=1680832473220Num uniques 1054; 2023-04-07 01:54:33,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-07 01:54:33,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680832473202, max timestamp=1680832473220Num uniques 1054; 2023-04-07 01:54:33,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680832473202, max timestamp=1680832473220Num uniques 1054; 2023-04-07 01:54:33,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-07 01:54:33,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-07 01:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832473257, max timestamp=1680832473272Num uniques 1063; 2023-04-07 01:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-07 01:54:33,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832473257, max timestamp=1680832473272Num uniques 1063; 2023-04-07 01:54:33,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680832473257, max timestamp=1680832473272Num uniques 1063; 2023-04-07 01:54:33,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-07 01:54:33,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-07 01:54:33,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832473308, max timestamp=1680832473322Num uniques 1072; 2023-04-07 01:54:33,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832473308, max timestamp=1680832473322Num uniques 1072; 2023-04-07 01:54:33,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 01:54:33,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 01:54:33,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832473308, max timestamp=1680832473322Num uniques 1072; 2023-04-07 01:54:33,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 01:54:33,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680832473350, max timestamp=1680832473362Num uniques 1081; 2023-04-07 01:54:33,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-07 01:54:33,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680832473350, max timestamp=1680832473362Num uniques 1081; 2023-04-07 01:54:33,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-07 01:54:33,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680832473350, max timestamp=1680832473362Num uniques 1081; 2023-04-07 01:54:33,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-07 01:54:33,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832533380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832533381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832533384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832533485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832533485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832533487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,507 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=752, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/8f1f6c3bccf14caf91acb7edcde143ea 2023-04-07 01:54:33,534 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/b4596194636a4a5da3ec83c4018acb4a 2023-04-07 01:54:33,568 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/65efeac5ebfc443790082dd18fc467b8 2023-04-07 01:54:33,574 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/8f1f6c3bccf14caf91acb7edcde143ea as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea 2023-04-07 01:54:33,579 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea, entries=150, sequenceid=752, filesize=25.4 K 2023-04-07 01:54:33,581 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/b4596194636a4a5da3ec83c4018acb4a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a 2023-04-07 01:54:33,587 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a, entries=150, sequenceid=752, filesize=11.9 K 2023-04-07 01:54:33,588 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/65efeac5ebfc443790082dd18fc467b8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8 2023-04-07 01:54:33,592 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8, entries=150, sequenceid=752, filesize=11.9 K 2023-04-07 01:54:33,593 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 9bfd0d326e03d54dad8a7c3526a41584 in 911ms, sequenceid=752, compaction requested=true 2023-04-07 01:54:33,594 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:33,594 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:33,599 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ed643f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:33,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:33,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,604 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:33,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:33,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:33,606 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:33,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:33,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:33,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:33,607 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:33,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:33,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:33,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:33,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:33,620 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:33,620 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:33,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:33,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=161.02 KB heapSize=449.25 KB 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:33,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:54:33,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832533700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832533702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832533703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,709 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407cfff4edaaeb94c9285bf4591321b43d6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfff4edaaeb94c9285bf4591321b43d6 2023-04-07 01:54:33,720 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:33,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:33,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:33,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:33,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:33,723 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:33,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:33,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:33,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:33,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,725 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:33,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,726 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,726 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:33,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:33,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. started... 2023-04-07 01:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. as already flushing 2023-04-07 01:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:33,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:33,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0788af[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:33,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,732 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:33,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:33,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:33,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:33,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:33,737 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:33,737 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:33,737 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:33,737 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:33,737 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:33,738 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,738 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:33,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,738 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:33,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:33,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:33,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:33,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:33,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:33,743 DEBUG [Thread-913] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:33,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:33,743 DEBUG [Thread-913] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:33,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,744 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,744 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:33,744 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:33,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:33,762 DEBUG [Thread-918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:61971 2023-04-07 01:54:33,762 DEBUG [Thread-918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:33,763 DEBUG [Thread-914] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:61971 2023-04-07 01:54:33,763 DEBUG [Thread-914] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:33,765 DEBUG [Thread-916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:61971 2023-04-07 01:54:33,765 DEBUG [Thread-916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:33,767 DEBUG [Thread-920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:61971 2023-04-07 01:54:33,767 DEBUG [Thread-920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:33,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832533805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832533805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832533804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48610 deadline: 1680832533842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,843 DEBUG [Thread-909] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:33,844 DEBUG [Thread-913] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:33,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:33,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48618 deadline: 1680832533871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:33,872 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:34,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832534006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832534007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832534007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,117 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=928, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/a56b65d1eb8246a7aadd9a5b1e4dac21 2023-04-07 01:54:34,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832534308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 497 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832534309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832534309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=928 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/3f3c8b6c430640eaa3f79157f7fba1c8 2023-04-07 01:54:34,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 499 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48662 deadline: 1680832534810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48632 deadline: 1680832534811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:34,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48636 deadline: 1680832534813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:34,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=928 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/a786c13220c64853957b83590235a734 2023-04-07 01:54:34,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/a56b65d1eb8246a7aadd9a5b1e4dac21 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21 2023-04-07 01:54:34,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21, entries=200, sequenceid=928, filesize=32.3 K 2023-04-07 01:54:34,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/3f3c8b6c430640eaa3f79157f7fba1c8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8 2023-04-07 01:54:34,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8, entries=150, sequenceid=928, filesize=11.9 K 2023-04-07 01:54:34,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/a786c13220c64853957b83590235a734 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734 2023-04-07 01:54:34,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734, entries=150, sequenceid=928, filesize=11.9 K 2023-04-07 01:54:34,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~449.20 KB/459984, currentSize=26.84 KB/27480 for 9bfd0d326e03d54dad8a7c3526a41584 in 1259ms, sequenceid=928, compaction requested=true 2023-04-07 01:54:34,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:34,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:34,954 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:34,954 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:34,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:34,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:34,955 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:34,955 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:34,955 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/A is initiating minor compaction (all files) 2023-04-07 01:54:34,956 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/B is initiating minor compaction (all files) 2023-04-07 01:54:34,956 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/A in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:34,956 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/B in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:34,956 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e9f36690d9f54c7586212784fd70a2d7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=109.4 K 2023-04-07 01:54:34,956 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b1f560c88b204ffbac34c4b6b9f87acf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=48.4 K 2023-04-07 01:54:34,956 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e9f36690d9f54c7586212784fd70a2d7, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:34,956 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b1f560c88b204ffbac34c4b6b9f87acf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:34,956 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5b7f1735ce194e8eaa05cc71c07e362a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1680832471223 2023-04-07 01:54:34,956 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d8e8184ca5f94ce48c2f36fd77f65d67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1680832471223 2023-04-07 01:54:34,957 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8f1f6c3bccf14caf91acb7edcde143ea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680832471824 2023-04-07 01:54:34,957 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b4596194636a4a5da3ec83c4018acb4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680832471824 2023-04-07 01:54:34,957 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a56b65d1eb8246a7aadd9a5b1e4dac21, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=928, earliestPutTs=1680832473378 2023-04-07 01:54:34,957 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3f3c8b6c430640eaa3f79157f7fba1c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=928, earliestPutTs=1680832473379 2023-04-07 01:54:34,966 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#B#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:34,968 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#A#compaction#198 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:35,381 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/e4caa837e7624a9c8b2456f40b4418f9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e4caa837e7624a9c8b2456f40b4418f9 2023-04-07 01:54:35,386 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/B of 9bfd0d326e03d54dad8a7c3526a41584 into e4caa837e7624a9c8b2456f40b4418f9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:35,386 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:35,386 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/B, priority=12, startTime=1680832474954; duration=0sec 2023-04-07 01:54:35,387 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:35,387 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:35,388 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:35,388 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 9bfd0d326e03d54dad8a7c3526a41584/C is initiating minor compaction (all files) 2023-04-07 01:54:35,388 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9bfd0d326e03d54dad8a7c3526a41584/C in TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:35,388 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/74bdf2f9e51a44ac816411197323caa2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp, totalSize=48.4 K 2023-04-07 01:54:35,388 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 74bdf2f9e51a44ac816411197323caa2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680832471069 2023-04-07 01:54:35,389 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1b32df062404498fa56fa0dfc9d105c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1680832471223 2023-04-07 01:54:35,389 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 65efeac5ebfc443790082dd18fc467b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680832471824 2023-04-07 01:54:35,389 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a786c13220c64853957b83590235a734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=928, earliestPutTs=1680832473379 2023-04-07 01:54:35,396 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 9bfd0d326e03d54dad8a7c3526a41584#C#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:35,804 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/9294804d86914e84b46f4159c83e8ef4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/9294804d86914e84b46f4159c83e8ef4 2023-04-07 01:54:35,808 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/C of 9bfd0d326e03d54dad8a7c3526a41584 into 9294804d86914e84b46f4159c83e8ef4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:35,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:35,808 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/C, priority=12, startTime=1680832474954; duration=0sec 2023-04-07 01:54:35,809 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:35,816 DEBUG [Thread-907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:61971 2023-04-07 01:54:35,816 DEBUG [Thread-907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:35,817 DEBUG [Thread-903] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:61971 2023-04-07 01:54:35,817 DEBUG [Thread-903] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:35,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:35,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:35,819 DEBUG [Thread-911] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:61971 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:35,819 DEBUG [Thread-911] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:35,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:36,192 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/c2bad0345be244c0a054e66fb21edf73 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/c2bad0345be244c0a054e66fb21edf73 2023-04-07 01:54:36,196 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9bfd0d326e03d54dad8a7c3526a41584/A of 9bfd0d326e03d54dad8a7c3526a41584 into c2bad0345be244c0a054e66fb21edf73(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-07 01:54:36,196 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:36,196 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584., storeName=9bfd0d326e03d54dad8a7c3526a41584/A, priority=12, startTime=1680832474954; duration=1sec 2023-04-07 01:54:36,197 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:36,231 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304070629b22c9fd54988af58bf8c300a879d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070629b22c9fd54988af58bf8c300a879d 2023-04-07 01:54:36,636 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=940, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/7f8022a2d93a4836b24830a656d6487c 2023-04-07 01:54:37,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/351fc143c3474930b38b010fd54c594f 2023-04-07 01:54:37,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/19261ff5c62a4a0aad8e9d1765af2687 2023-04-07 01:54:37,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/7f8022a2d93a4836b24830a656d6487c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7f8022a2d93a4836b24830a656d6487c 2023-04-07 01:54:37,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7f8022a2d93a4836b24830a656d6487c, entries=150, sequenceid=940, filesize=25.4 K 2023-04-07 01:54:37,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/351fc143c3474930b38b010fd54c594f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/351fc143c3474930b38b010fd54c594f 2023-04-07 01:54:37,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/351fc143c3474930b38b010fd54c594f, entries=150, sequenceid=940, filesize=11.9 K 2023-04-07 01:54:37,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/19261ff5c62a4a0aad8e9d1765af2687 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/19261ff5c62a4a0aad8e9d1765af2687 2023-04-07 01:54:37,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/19261ff5c62a4a0aad8e9d1765af2687, entries=150, sequenceid=940, filesize=11.9 K 2023-04-07 01:54:37,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 9bfd0d326e03d54dad8a7c3526a41584 in 1653ms, sequenceid=940, compaction requested=false 2023-04-07 01:54:37,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:43,876 DEBUG [Thread-905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:61971 2023-04-07 01:54:43,876 DEBUG [Thread-905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:43,943 DEBUG [Thread-909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:61971 2023-04-07 01:54:43,943 DEBUG [Thread-909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 404 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 288 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4878 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4813 2023-04-07 01:54:43,943 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:54:43,944 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2039 2023-04-07 01:54:43,944 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6117 rows 2023-04-07 01:54:43,944 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2042 2023-04-07 01:54:43,944 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6126 rows 2023-04-07 01:54:43,944 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:54:43,944 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:61971 2023-04-07 01:54:43,944 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:54:43,946 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:54:43,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:54:43,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:43,953 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832483952"}]},"ts":"1680832483952"} 2023-04-07 01:54:43,954 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:54:43,955 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:54:43,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, UNASSIGN}] 2023-04-07 01:54:43,957 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, UNASSIGN 2023-04-07 01:54:43,958 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:43,958 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832483958"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832483958"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832483958"}]},"ts":"1680832483958"} 2023-04-07 01:54:43,959 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:44,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:44,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9bfd0d326e03d54dad8a7c3526a41584, disabling compactions & flushes 2023-04-07 01:54:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. after waiting 0 ms 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9bfd0d326e03d54dad8a7c3526a41584 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=A 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=B 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9bfd0d326e03d54dad8a7c3526a41584, store=C 2023-04-07 01:54:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:44,313 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-07 01:54:44,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407ff9b7e0543964bab9763e94acab0c5cb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ff9b7e0543964bab9763e94acab0c5cb 2023-04-07 01:54:44,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:44,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=946, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/fd98a28d79de4ea7846fbf5c8a8b8111 2023-04-07 01:54:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:45,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=946 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/cae9162580b3486f8081faa7dffb68bd 2023-04-07 01:54:45,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=946 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/a798999f13a4444aa0e2c63c3cf2db4e 2023-04-07 01:54:45,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/A/fd98a28d79de4ea7846fbf5c8a8b8111 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fd98a28d79de4ea7846fbf5c8a8b8111 2023-04-07 01:54:45,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fd98a28d79de4ea7846fbf5c8a8b8111, entries=50, sequenceid=946, filesize=11.7 K 2023-04-07 01:54:45,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/B/cae9162580b3486f8081faa7dffb68bd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/cae9162580b3486f8081faa7dffb68bd 2023-04-07 01:54:45,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/cae9162580b3486f8081faa7dffb68bd, entries=50, sequenceid=946, filesize=7.1 K 2023-04-07 01:54:45,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/.tmp/C/a798999f13a4444aa0e2c63c3cf2db4e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a798999f13a4444aa0e2c63c3cf2db4e 2023-04-07 01:54:45,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a798999f13a4444aa0e2c63c3cf2db4e, entries=50, sequenceid=946, filesize=7.1 K 2023-04-07 01:54:45,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 9bfd0d326e03d54dad8a7c3526a41584 in 1654ms, sequenceid=946, compaction requested=true 2023-04-07 01:54:45,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8bab34bac53442d48f82e406fc0f0160, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/56e6b508c8cb47edb9f86371f54b6880, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce57848b0e81421bbfdc903e153730f7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/38248deb3a2449eab9f1cc26a0222666, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/765d0b90241b47f2a99968be7187c8ff, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e9f36690d9f54c7586212784fd70a2d7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21] to archive 2023-04-07 01:54:45,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:45,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a134764e2f57445a80923763eb6abb5e 2023-04-07 01:54:45,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/98632fe3309c4ca28103ba0929334723 2023-04-07 01:54:45,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8bab34bac53442d48f82e406fc0f0160 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8bab34bac53442d48f82e406fc0f0160 2023-04-07 01:54:45,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/d1d8d2641b9f455890a38339dec8251c 2023-04-07 01:54:45,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3a188bc986fa45d6ace3e631bf3cac87 2023-04-07 01:54:45,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/56e6b508c8cb47edb9f86371f54b6880 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/56e6b508c8cb47edb9f86371f54b6880 2023-04-07 01:54:45,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/885804b629b54a0f8d16174fe8135267 2023-04-07 01:54:45,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/96e8e3f3b1fe4829aa0223bd19a1e7b5 2023-04-07 01:54:45,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/9822d6543cdc4d9da50da471acdbaa3b 2023-04-07 01:54:45,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/97cde93c85934d96ac564778773824e2 2023-04-07 01:54:45,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/87f280076c7a46e1b2af52a4d3e7caac 2023-04-07 01:54:45,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce57848b0e81421bbfdc903e153730f7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce57848b0e81421bbfdc903e153730f7 2023-04-07 01:54:45,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e1a2923902584437989f8c5648b84baa 2023-04-07 01:54:45,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7c4b36cda0fe49fc884bb85df634ec7b 2023-04-07 01:54:45,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/38248deb3a2449eab9f1cc26a0222666 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/38248deb3a2449eab9f1cc26a0222666 2023-04-07 01:54:45,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/60a499c91c434575880892b14cb3fd5e 2023-04-07 01:54:45,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/ce59466fbe2e476d9c897d575bc36a17 2023-04-07 01:54:45,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/f1d85e148af347d3b5a96e7e35046e56 2023-04-07 01:54:45,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/765d0b90241b47f2a99968be7187c8ff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/765d0b90241b47f2a99968be7187c8ff 2023-04-07 01:54:45,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/769e248df9c44d0eb7fe58d463a456b2 2023-04-07 01:54:45,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/16f203a3a7c34470a14391187fab1a06 2023-04-07 01:54:45,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/3c8ab13119914f74bd66365fd9fe80ee 2023-04-07 01:54:45,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fae78cc2a46e4a509a8a4175764fd441 2023-04-07 01:54:45,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e9f36690d9f54c7586212784fd70a2d7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/e9f36690d9f54c7586212784fd70a2d7 2023-04-07 01:54:45,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/5b7f1735ce194e8eaa05cc71c07e362a 2023-04-07 01:54:45,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/8f1f6c3bccf14caf91acb7edcde143ea 2023-04-07 01:54:45,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/a56b65d1eb8246a7aadd9a5b1e4dac21 2023-04-07 01:54:45,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/a1b44857127343108903f21c2f204607, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0d0667ad81ec45cb9f8e21d6b97e3021, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/564d6cd4be524d108c0508d2e0ca7861, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/02609460a8f1476ca2e04996f60d369f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3fa14ec73fbb445bb2ac41383b90eda6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b1f560c88b204ffbac34c4b6b9f87acf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8] to archive 2023-04-07 01:54:45,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:45,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/5eaacb92e5bb40098f2d970beab63e93 2023-04-07 01:54:45,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/fface081e1f3406f8ae3bb4413f0e93a 2023-04-07 01:54:45,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/a1b44857127343108903f21c2f204607 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/a1b44857127343108903f21c2f204607 2023-04-07 01:54:45,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/7e0acd8f6a41404a96042131329e079e 2023-04-07 01:54:45,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/bc97ce6c155c48b3a39233718e45bc6f 2023-04-07 01:54:45,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0d0667ad81ec45cb9f8e21d6b97e3021 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0d0667ad81ec45cb9f8e21d6b97e3021 2023-04-07 01:54:45,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/6cdd772ab288483c9afbc8b48d16123d 2023-04-07 01:54:45,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/65ec605634c24342967d2d290184303f 2023-04-07 01:54:45,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/47179463e513404f8a4cf0801b520a5e 2023-04-07 01:54:45,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0b5481465e5642308082d21777322fa3 2023-04-07 01:54:45,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/564d6cd4be524d108c0508d2e0ca7861 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/564d6cd4be524d108c0508d2e0ca7861 2023-04-07 01:54:45,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/968c4aef8f124bf58e8524f69b443598 2023-04-07 01:54:45,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9d790e29ee3c4194990bf2981158fe7c 2023-04-07 01:54:45,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/02609460a8f1476ca2e04996f60d369f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/02609460a8f1476ca2e04996f60d369f 2023-04-07 01:54:45,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/c03ed4f6d37e4f0e92516d5d67a79c7d 2023-04-07 01:54:45,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e8fa39f0b8d44192bdc96c9ee9acbc98 2023-04-07 01:54:45,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/73503a33b43949aaa8e7cecf8f4f2f8d 2023-04-07 01:54:45,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/59c7b9aa642a4bbcb43caa5747568975 2023-04-07 01:54:45,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3fa14ec73fbb445bb2ac41383b90eda6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3fa14ec73fbb445bb2ac41383b90eda6 2023-04-07 01:54:45,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/9412a22ab6754d7abc2ee51ffa46ed8e 2023-04-07 01:54:45,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/0a3f416d798f49959a644500f762daf3 2023-04-07 01:54:45,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/057e5ad240f5444f84442ccb7eab9c9a 2023-04-07 01:54:45,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b1f560c88b204ffbac34c4b6b9f87acf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b1f560c88b204ffbac34c4b6b9f87acf 2023-04-07 01:54:45,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/dd7773765cd94c9b82cd4d2d07c65a92 2023-04-07 01:54:45,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/d8e8184ca5f94ce48c2f36fd77f65d67 2023-04-07 01:54:45,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/b4596194636a4a5da3ec83c4018acb4a 2023-04-07 01:54:45,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/3f3c8b6c430640eaa3f79157f7fba1c8 2023-04-07 01:54:45,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/ce2cea476a5c46e0a445baa5f5844aa0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30f699eece8c4de59d0eae48eada088f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/6f7b4ea949094b3c8822915d82c25b89, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/374bf860d3d347eab7d307bb266dbb49, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/512101e32b5d454f9ef39b091ab596bc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/74bdf2f9e51a44ac816411197323caa2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734] to archive 2023-04-07 01:54:45,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:54:45,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/653b8c462fdf49fb9eae575a4550b905 2023-04-07 01:54:45,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5d2ce0802f174ffdbb834dc35336d2e2 2023-04-07 01:54:45,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/ce2cea476a5c46e0a445baa5f5844aa0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/ce2cea476a5c46e0a445baa5f5844aa0 2023-04-07 01:54:45,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/59553f2a1b8e47a790554d51fc567b55 2023-04-07 01:54:45,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d0aff4b7e1bd4c28ab6d58da7e764498 2023-04-07 01:54:45,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30f699eece8c4de59d0eae48eada088f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30f699eece8c4de59d0eae48eada088f 2023-04-07 01:54:45,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/d88a8844d3f3403590c098afe8b7225c 2023-04-07 01:54:45,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/4a16b2692ca347a3911d1062c44ca8b1 2023-04-07 01:54:45,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/0ee121acb51b4113879324dafb26b152 2023-04-07 01:54:45,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/13e8a8d1629943b89e63f6b8777b2b88 2023-04-07 01:54:45,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/6f7b4ea949094b3c8822915d82c25b89 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/6f7b4ea949094b3c8822915d82c25b89 2023-04-07 01:54:45,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/3c0233007c2448a083404f50b892d8a3 2023-04-07 01:54:45,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/7d132cf155974cc0bb2a0c8495af537e 2023-04-07 01:54:45,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/374bf860d3d347eab7d307bb266dbb49 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/374bf860d3d347eab7d307bb266dbb49 2023-04-07 01:54:45,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/30d355b663304b8d88c759607bd291e3 2023-04-07 01:54:45,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/b14e1717c2c1460a9c652509916f1ceb 2023-04-07 01:54:45,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/5a04a21d47e34775820adae4a7b1c697 2023-04-07 01:54:45,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/c59f5d1127b84a6288b66ffa1c20c1db 2023-04-07 01:54:45,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/512101e32b5d454f9ef39b091ab596bc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/512101e32b5d454f9ef39b091ab596bc 2023-04-07 01:54:45,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/df92165f13844f4cbbc6c3c7dcd5e19a 2023-04-07 01:54:45,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/52da7dd870254096ac84a93b723cc7b6 2023-04-07 01:54:45,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/f75bf3ebe417488597109c325511f81c 2023-04-07 01:54:45,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/74bdf2f9e51a44ac816411197323caa2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/74bdf2f9e51a44ac816411197323caa2 2023-04-07 01:54:45,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/2412e818a01f461c8d4f01c2edf68793 2023-04-07 01:54:45,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/1b32df062404498fa56fa0dfc9d105c1 2023-04-07 01:54:45,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/65efeac5ebfc443790082dd18fc467b8 2023-04-07 01:54:45,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a786c13220c64853957b83590235a734 2023-04-07 01:54:45,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits/949.seqid, newMaxSeqId=949, maxSeqId=4 2023-04-07 01:54:45,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584. 2023-04-07 01:54:45,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9bfd0d326e03d54dad8a7c3526a41584: 2023-04-07 01:54:45,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:45,891 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9bfd0d326e03d54dad8a7c3526a41584, regionState=CLOSED 2023-04-07 01:54:45,891 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832485891"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832485891"}]},"ts":"1680832485891"} 2023-04-07 01:54:45,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-07 01:54:45,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 9bfd0d326e03d54dad8a7c3526a41584, server=jenkins-hbase4.apache.org,44961,1680832421898 in 1.9330 sec 2023-04-07 01:54:45,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-07 01:54:45,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9bfd0d326e03d54dad8a7c3526a41584, UNASSIGN in 1.9370 sec 2023-04-07 01:54:45,895 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832485895"}]},"ts":"1680832485895"} 2023-04-07 01:54:45,896 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:54:45,897 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:54:45,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9510 sec 2023-04-07 01:54:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 01:54:46,057 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-07 01:54:46,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:54:46,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,060 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-07 01:54:46,061 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,067 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:46,071 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits] 2023-04-07 01:54:46,075 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7f8022a2d93a4836b24830a656d6487c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/7f8022a2d93a4836b24830a656d6487c 2023-04-07 01:54:46,076 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/c2bad0345be244c0a054e66fb21edf73 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/c2bad0345be244c0a054e66fb21edf73 2023-04-07 01:54:46,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fd98a28d79de4ea7846fbf5c8a8b8111 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/A/fd98a28d79de4ea7846fbf5c8a8b8111 2023-04-07 01:54:46,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/351fc143c3474930b38b010fd54c594f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/351fc143c3474930b38b010fd54c594f 2023-04-07 01:54:46,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/cae9162580b3486f8081faa7dffb68bd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/cae9162580b3486f8081faa7dffb68bd 2023-04-07 01:54:46,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e4caa837e7624a9c8b2456f40b4418f9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/B/e4caa837e7624a9c8b2456f40b4418f9 2023-04-07 01:54:46,088 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/19261ff5c62a4a0aad8e9d1765af2687 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/19261ff5c62a4a0aad8e9d1765af2687 2023-04-07 01:54:46,093 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/9294804d86914e84b46f4159c83e8ef4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/9294804d86914e84b46f4159c83e8ef4 2023-04-07 01:54:46,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a798999f13a4444aa0e2c63c3cf2db4e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/C/a798999f13a4444aa0e2c63c3cf2db4e 2023-04-07 01:54:46,098 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits/949.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584/recovered.edits/949.seqid 2023-04-07 01:54:46,099 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/9bfd0d326e03d54dad8a7c3526a41584 2023-04-07 01:54:46,099 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:54:46,100 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:54:46,101 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 01:54:46,106 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407052619afc80047a8a8cd25302d887b13 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407052619afc80047a8a8cd25302d887b13 2023-04-07 01:54:46,107 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070629b22c9fd54988af58bf8c300a879d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070629b22c9fd54988af58bf8c300a879d 2023-04-07 01:54:46,109 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f7930c5f341494ea08a66b6472cc150 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f7930c5f341494ea08a66b6472cc150 2023-04-07 01:54:46,110 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719688416b006421e9b57ee19fe695634 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719688416b006421e9b57ee19fe695634 2023-04-07 01:54:46,111 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071dbfc136262f4022abd1ed70ddcd0dff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071dbfc136262f4022abd1ed70ddcd0dff 2023-04-07 01:54:46,112 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075f8b0851fb144dbfa03c5fcc0f097f96 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075f8b0851fb144dbfa03c5fcc0f097f96 2023-04-07 01:54:46,113 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076aa63bd414b741cba269880ccf36896e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076aa63bd414b741cba269880ccf36896e 2023-04-07 01:54:46,114 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040774a9bf7a43044068baa64553c67e0332 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040774a9bf7a43044068baa64553c67e0332 2023-04-07 01:54:46,115 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040776164e828ada41b7a38bdf0b4ee4391a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040776164e828ada41b7a38bdf0b4ee4391a 2023-04-07 01:54:46,117 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077943e9183e8247bf8052dfd20e802231 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077943e9183e8247bf8052dfd20e802231 2023-04-07 01:54:46,118 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079699dc975bdb4f4ab57bb380378aab2e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079699dc975bdb4f4ab57bb380378aab2e 2023-04-07 01:54:46,120 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407acfa3c914052477294d32125b66dac07 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407acfa3c914052477294d32125b66dac07 2023-04-07 01:54:46,121 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b494f1fa4b184ef39d3e9bfe351950e4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b494f1fa4b184ef39d3e9bfe351950e4 2023-04-07 01:54:46,122 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407babbe4d0b8234e5fb04aa8161fab5e84 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407babbe4d0b8234e5fb04aa8161fab5e84 2023-04-07 01:54:46,123 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c6adf72af95d4775a135de78e8e31f57 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c6adf72af95d4775a135de78e8e31f57 2023-04-07 01:54:46,124 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c7eec748d2784d2386f9e0d56f0be21b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c7eec748d2784d2386f9e0d56f0be21b 2023-04-07 01:54:46,125 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfff4edaaeb94c9285bf4591321b43d6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfff4edaaeb94c9285bf4591321b43d6 2023-04-07 01:54:46,126 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d2317555eb4446b7adf2e112e3cc2037 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d2317555eb4446b7adf2e112e3cc2037 2023-04-07 01:54:46,127 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d6b1b698b7964825a8de8999824ed718 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d6b1b698b7964825a8de8999824ed718 2023-04-07 01:54:46,129 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e4184fd2e9914b3d99444e4690e85cb7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e4184fd2e9914b3d99444e4690e85cb7 2023-04-07 01:54:46,130 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eeccb76e27294dbeafaee0e3045cab99 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eeccb76e27294dbeafaee0e3045cab99 2023-04-07 01:54:46,131 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fa3e77a2a0a14bd4904a9cfc9483b56f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fa3e77a2a0a14bd4904a9cfc9483b56f 2023-04-07 01:54:46,132 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ff9b7e0543964bab9763e94acab0c5cb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ff9b7e0543964bab9763e94acab0c5cb 2023-04-07 01:54:46,133 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:54:46,135 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,137 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:54:46,139 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:54:46,140 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,140 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:54:46,140 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832486140"}]},"ts":"9223372036854775807"} 2023-04-07 01:54:46,146 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:54:46,146 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9bfd0d326e03d54dad8a7c3526a41584, NAME => 'TestAcidGuarantees,,1680832450614.9bfd0d326e03d54dad8a7c3526a41584.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:54:46,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:54:46,146 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832486146"}]},"ts":"9223372036854775807"} 2023-04-07 01:54:46,148 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:54:46,151 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 93 msec 2023-04-07 01:54:46,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-07 01:54:46,161 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-07 01:54:46,174 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=226 (was 234), OpenFileDescriptor=520 (was 568), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 398), ProcessCount=169 (was 169), AvailableMemoryMB=2739 (was 2891) 2023-04-07 01:54:46,187 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226, OpenFileDescriptor=520, MaxFileDescriptor=60000, SystemLoadAverage=371, ProcessCount=169, AvailableMemoryMB=2738 2023-04-07 01:54:46,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:54:46,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:54:46,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:54:46,192 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:54:46,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-07 01:54:46,192 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:54:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 01:54:46,194 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,194 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb empty. 2023-04-07 01:54:46,194 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,194 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:54:46,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 01:54:46,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 01:54:46,603 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:54:46,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ff332f26550df115caee283ff76382bb, NAME => 'TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:54:46,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:54:46,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ff332f26550df115caee283ff76382bb, disabling compactions & flushes 2023-04-07 01:54:46,613 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. after waiting 0 ms 2023-04-07 01:54:46,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,613 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:46,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:54:46,616 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832486616"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832486616"}]},"ts":"1680832486616"} 2023-04-07 01:54:46,617 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:54:46,618 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:54:46,618 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832486618"}]},"ts":"1680832486618"} 2023-04-07 01:54:46,619 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:54:46,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, ASSIGN}] 2023-04-07 01:54:46,623 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, ASSIGN 2023-04-07 01:54:46,623 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:54:46,774 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ff332f26550df115caee283ff76382bb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:46,775 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832486774"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832486774"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832486774"}]},"ts":"1680832486774"} 2023-04-07 01:54:46,776 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:54:46,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 01:54:46,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ff332f26550df115caee283ff76382bb, NAME => 'TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:54:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:54:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,934 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,936 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A 2023-04-07 01:54:46,936 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A 2023-04-07 01:54:46,936 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:46,937 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff332f26550df115caee283ff76382bb columnFamilyName A 2023-04-07 01:54:46,938 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(310): Store=ff332f26550df115caee283ff76382bb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:46,938 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,941 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B 2023-04-07 01:54:46,941 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B 2023-04-07 01:54:46,941 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:46,941 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff332f26550df115caee283ff76382bb columnFamilyName B 2023-04-07 01:54:46,942 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(310): Store=ff332f26550df115caee283ff76382bb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:46,942 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,943 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C 2023-04-07 01:54:46,944 DEBUG [StoreOpener-ff332f26550df115caee283ff76382bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C 2023-04-07 01:54:46,944 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:54:46,944 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff332f26550df115caee283ff76382bb columnFamilyName C 2023-04-07 01:54:46,944 INFO [StoreOpener-ff332f26550df115caee283ff76382bb-1] regionserver.HStore(310): Store=ff332f26550df115caee283ff76382bb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:54:46,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:54:46,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ff332f26550df115caee283ff76382bb 2023-04-07 01:54:46,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:54:46,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ff332f26550df115caee283ff76382bb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62572099, jitterRate=-0.06760306656360626}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:54:46,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:46,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., pid=30, masterSystemTime=1680832486927 2023-04-07 01:54:46,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:46,955 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ff332f26550df115caee283ff76382bb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:46,955 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832486955"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832486955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832486955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832486955"}]},"ts":"1680832486955"} 2023-04-07 01:54:46,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-07 01:54:46,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 in 180 msec 2023-04-07 01:54:46,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-07 01:54:46,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, ASSIGN in 336 msec 2023-04-07 01:54:46,960 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:54:46,960 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832486960"}]},"ts":"1680832486960"} 2023-04-07 01:54:46,961 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:54:46,964 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:54:46,966 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 774 msec 2023-04-07 01:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 01:54:47,296 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-07 01:54:47,297 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,303 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,304 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,305 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,307 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:54:47,309 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:54:47,310 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,314 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,314 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,318 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,319 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,324 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,324 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,328 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,329 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,333 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,333 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,338 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,338 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,344 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,344 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,350 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,351 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,356 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,356 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:54:47,360 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:54:47,364 DEBUG [hconnection-0x73e4c3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,364 DEBUG [hconnection-0x748d049b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,365 DEBUG [hconnection-0x15114b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,365 DEBUG [hconnection-0x1dbc5155-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,365 DEBUG [hconnection-0x6e8cefea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,365 DEBUG [hconnection-0x3039caef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:47,366 DEBUG [hconnection-0x480b2248-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,367 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,367 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,371 DEBUG [hconnection-0x7733738a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,372 DEBUG [hconnection-0x2fa9d101-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,372 DEBUG [hconnection-0x168a64f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:54:47,373 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,373 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,373 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,373 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60734, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,375 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60796, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,376 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:47,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:47,381 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:47,381 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,381 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:47,381 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:47,382 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:54:47,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,386 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:47,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:47,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:47,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:47,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:47,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:47,392 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,392 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:47,392 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:47,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:47,396 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470ae09d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:47,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:47,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,403 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:47,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:47,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:47,408 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:47,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:47,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:47,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:47,408 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:47,411 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,411 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,411 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832487404, max timestamp=1680832487410Num uniques 859; 2023-04-07 01:54:47,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:47,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832487404, max timestamp=1680832487410Num uniques 859; 2023-04-07 01:54:47,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832487404, max timestamp=1680832487410Num uniques 859; 2023-04-07 01:54:47,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:47,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:47,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:47,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,420 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:47,420 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:47,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832547427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832547427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832547427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832547428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832547429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,520 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:47,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:47,524 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:47,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:47,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:47,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,526 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:47,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:47,529 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,529 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:47,529 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:47,529 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:47,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:47,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b2dc0d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832547531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832547531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832547531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832547532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:47,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832547533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:47,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,537 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:47,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:47,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:47,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:47,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:47,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:47,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,546 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:47,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:47,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,546 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,547 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:47,547 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,547 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:47,547 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,647 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:47,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:47,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:47,651 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:47,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:47,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:47,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,654 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,654 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:47,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:47,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:47,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,657 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f19198[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:47,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:47,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,668 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:47,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:47,670 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:47,673 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:47,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:47,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:47,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:47,673 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:47,675 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,675 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,675 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,681 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,681 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:47,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:47,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:47,682 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,682 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:47,682 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:47,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832547734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832547734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832547736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832547738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:47,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832547743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,782 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:47,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:47,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:47,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:47,786 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:47,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:47,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:47,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,787 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,787 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:47,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:47,789 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,789 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:47,789 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:47,789 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:47,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d32f19[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:47,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:47,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,794 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:47,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:47,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:47,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:47,799 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:47,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:47,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:47,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:47,800 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:47,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:47,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:47,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,819 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:47,819 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:47,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:47,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:47,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/98b7b77f0a6848baa5282f94c0c6f45f 2023-04-07 01:54:47,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bd5fcbf1f8654e82bced1edf56aa95fa 2023-04-07 01:54:47,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/37ad0f9019b4426fa7647d8dd3aa4309 2023-04-07 01:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/98b7b77f0a6848baa5282f94c0c6f45f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f 2023-04-07 01:54:47,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:54:47,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bd5fcbf1f8654e82bced1edf56aa95fa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa 2023-04-07 01:54:47,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:54:47,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/37ad0f9019b4426fa7647d8dd3aa4309 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309 2023-04-07 01:54:47,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 01:54:47,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for ff332f26550df115caee283ff76382bb in 495ms, sequenceid=12, compaction requested=false 2023-04-07 01:54:47,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:47,920 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:47,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:47,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:47,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:47,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:47,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:47,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:47,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:47,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:47,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:47,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:47,928 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:47,928 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:47,942 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/9c10a300f3394e77baf59b819c358f59 2023-04-07 01:54:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:48,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:48,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832548044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832548045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832548047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832548047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832548048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832548149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832548149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832548151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832548152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,367 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/318ffe69509741259d4e47bceefef4cf 2023-04-07 01:54:48,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832548367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832548367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832548367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832548367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,377 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/26652907d8c4435598979dc58d82caf8 2023-04-07 01:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/9c10a300f3394e77baf59b819c358f59 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59 2023-04-07 01:54:48,385 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 01:54:48,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/318ffe69509741259d4e47bceefef4cf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf 2023-04-07 01:54:48,390 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 01:54:48,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/26652907d8c4435598979dc58d82caf8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8 2023-04-07 01:54:48,395 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 01:54:48,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for ff332f26550df115caee283ff76382bb in 468ms, sequenceid=47, compaction requested=false 2023-04-07 01:54:48,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:48,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:48,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164d2956[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:48,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:48,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,402 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:48,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:48,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:48,404 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:48,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:48,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:48,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:48,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:48,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:48,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:48,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:48,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:48,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,414 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:48,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:48,414 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:48,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:48,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:48,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:48,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,514 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:48,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:48,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:48,517 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:48,517 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:48,517 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:48,517 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,519 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:48,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:48,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:48,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:48,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:48,521 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:48,521 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:48,530 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/2bd6f7d70eb743c280962eb3b298f4f1 2023-04-07 01:54:48,538 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/d90aa4d8bb684c01a7a2b7381f53dff3 2023-04-07 01:54:48,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:48,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:48,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832488598, max timestamp=1680832488607Num uniques 859; 2023-04-07 01:54:48,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:48,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832488598, max timestamp=1680832488607Num uniques 859; 2023-04-07 01:54:48,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832488598, max timestamp=1680832488607Num uniques 859; 2023-04-07 01:54:48,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:48,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:48,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832548659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832548670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832548670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832548671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832548671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832548764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,948 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/964cc1178399463fb88c98152e38da33 2023-04-07 01:54:48,952 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/2bd6f7d70eb743c280962eb3b298f4f1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1 2023-04-07 01:54:48,956 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1, entries=150, sequenceid=58, filesize=11.6 K 2023-04-07 01:54:48,957 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/d90aa4d8bb684c01a7a2b7381f53dff3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3 2023-04-07 01:54:48,963 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3, entries=150, sequenceid=58, filesize=11.6 K 2023-04-07 01:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/964cc1178399463fb88c98152e38da33 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33 2023-04-07 01:54:48,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:48,972 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33, entries=150, sequenceid=58, filesize=11.6 K 2023-04-07 01:54:48,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832548965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,973 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ff332f26550df115caee283ff76382bb in 452ms, sequenceid=58, compaction requested=true 2023-04-07 01:54:48,973 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:48,973 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:48,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6960dd26[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:48,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:48,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:48,979 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,979 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:48,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:48,991 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:48,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:48,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:48,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:48,991 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:48,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:48,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:48,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:48,993 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,994 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,996 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,996 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:48,996 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,996 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:48,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:48,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:48,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:48,996 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:48,996 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:48,997 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:48,997 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:48,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:48,997 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,997 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:48,997 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:48,997 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,097 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:49,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:49,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:49,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:49,100 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:49,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:49,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:49,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:49,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:49,103 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:49,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:49,103 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,103 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:49,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:49,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:49,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:49,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:49,105 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:49,105 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:49,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832549181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832549183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832549183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832549185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832549275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832549287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832549287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832549287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832549290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832549490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832549490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832549491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832549495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,515 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/99c257c5e7944a4cab687c7e7917cbca 2023-04-07 01:54:49,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832549781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832549793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832549794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832549795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832549798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,928 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/890ce935ca0c46338d1eb81f8862c10d 2023-04-07 01:54:49,940 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/411bc8541bdd4330b6c97a747dad9420 2023-04-07 01:54:49,944 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/99c257c5e7944a4cab687c7e7917cbca as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca 2023-04-07 01:54:49,947 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca, entries=150, sequenceid=94, filesize=11.6 K 2023-04-07 01:54:49,948 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/890ce935ca0c46338d1eb81f8862c10d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d 2023-04-07 01:54:49,956 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d, entries=150, sequenceid=94, filesize=11.6 K 2023-04-07 01:54:49,956 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/411bc8541bdd4330b6c97a747dad9420 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420 2023-04-07 01:54:49,960 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420, entries=150, sequenceid=94, filesize=11.6 K 2023-04-07 01:54:49,960 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for ff332f26550df115caee283ff76382bb in 855ms, sequenceid=94, compaction requested=true 2023-04-07 01:54:49,960 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:49,960 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:49,960 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:49,962 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7ecc3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:49,965 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:49,965 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,967 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:49,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:49,967 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:49,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:49,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:49,971 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:49,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:49,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:49,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:49,971 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:49,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:49,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:49,974 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:49,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:49,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:49,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:49,979 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:49,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:49,979 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:49,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:49,979 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:49,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:49,979 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:49,979 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:49,979 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:49,979 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:49,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,079 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:50,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:50,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:50,083 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:50,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:50,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:50,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,085 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:50,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:50,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:50,087 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:50,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:50,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:50,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:50,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:50,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:50,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:50,100 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/894f693ded1b4774a450be8997edbd77 2023-04-07 01:54:50,123 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/e0dc28e42fcf448abf7000d0cdf3823c 2023-04-07 01:54:50,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:50,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:50,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832550316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832490313, max timestamp=1680832490320Num uniques 859; 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832490313, max timestamp=1680832490320Num uniques 859; 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832490313, max timestamp=1680832490320Num uniques 859; 2023-04-07 01:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:50,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832550340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832550341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832550342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832550423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832550443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832550445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832550445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,533 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/497e46c1ccd641259b1187d2bd72e8c0 2023-04-07 01:54:50,537 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/894f693ded1b4774a450be8997edbd77 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77 2023-04-07 01:54:50,541 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77, entries=150, sequenceid=105, filesize=11.6 K 2023-04-07 01:54:50,541 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/e0dc28e42fcf448abf7000d0cdf3823c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c 2023-04-07 01:54:50,547 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c, entries=150, sequenceid=105, filesize=11.6 K 2023-04-07 01:54:50,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/497e46c1ccd641259b1187d2bd72e8c0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0 2023-04-07 01:54:50,557 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0, entries=150, sequenceid=105, filesize=11.6 K 2023-04-07 01:54:50,558 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ff332f26550df115caee283ff76382bb in 471ms, sequenceid=105, compaction requested=true 2023-04-07 01:54:50,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:50,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc3a043[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:50,565 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:50,565 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:50,568 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,568 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:50,569 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:50,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:50,572 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:50,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:50,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:50,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:50,572 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:50,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:50,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,578 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:50,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,578 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:50,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:50,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:50,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832550648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832550649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832550649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832550652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,678 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:50,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:50,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:50,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:50,683 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:50,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:50,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:50,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,683 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:50,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:50,687 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,687 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:50,687 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:50,687 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:50,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:50,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:50,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633b2d14[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:50,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,692 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:50,692 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,692 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:50,694 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,694 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:50,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:50,696 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:50,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:50,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:50,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:50,696 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:50,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:50,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:50,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:50,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,705 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,705 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832550753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832550756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832550792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,805 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:50,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:50,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:50,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:50,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:50,809 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:50,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:50,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:50,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:50,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:50,812 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,812 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:50,812 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:50,812 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d45b8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:50,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:50,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,817 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:50,818 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:50,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:50,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:50,821 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:50,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:50,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:50,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:50,821 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:50,823 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,823 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,823 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:50,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:50,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:50,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,828 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:50,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,828 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:50,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,828 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,928 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:50,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:50,931 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:50,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:50,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:50,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:50,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:50,936 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,936 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:50,936 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:50,936 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:50,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30703104[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:50,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,941 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:50,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:50,946 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:50,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:50,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:50,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:50,946 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:50,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:50,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:50,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:50,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:50,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:50,953 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:50,953 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:50,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832550952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832550955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832550956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:50,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:50,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832550961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/817ce063d5374b61a2ea347cc1403e46 2023-04-07 01:54:51,053 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:51,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:51,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:51,056 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:51,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:51,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:51,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:51,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:51,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:51,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:51,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:51,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31304ef1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:51,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:51,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:51,068 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:51,070 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:51,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:51,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:51,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:51,070 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:51,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:51,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,081 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,081 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,182 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:51,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:51,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:51,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:51,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:51,184 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:51,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:51,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:51,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:51,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:51,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:51,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:51,188 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:51,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:51,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd9864e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:51,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:51,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,193 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:51,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:51,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:51,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:51,197 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:51,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:51,197 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:51,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:51,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,203 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:51,203 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,204 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,204 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832551259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832551266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,303 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:51,307 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,309 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:51,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:51,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:51,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:51,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:51,313 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dcf7208[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:51,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:51,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,316 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:51,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:51,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:51,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:51,318 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:51,320 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:51,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:51,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:51,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:51,320 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:51,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:51,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:51,327 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,327 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:51,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,327 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:51,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:51,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,420 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:54:51,427 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:51,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:51,431 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:51,431 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:51,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:51,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:51,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:51,441 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8696a4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:51,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:51,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,446 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:51,446 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:51,446 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:51,446 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,446 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:51,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:51,450 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:51,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:51,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:51,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:51,450 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:51,452 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,452 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a7cba3789ed4d0a8ccc7ae33f2509c7 2023-04-07 01:54:51,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,453 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,453 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,455 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832551455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,458 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:51,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:51,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:51,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:51,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:51,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:51,460 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:51,460 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:51,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832551459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/1e64dc83b95148748d40c6d7e9bfbf00 2023-04-07 01:54:51,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/817ce063d5374b61a2ea347cc1403e46 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46 2023-04-07 01:54:51,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46, entries=200, sequenceid=140, filesize=14.1 K 2023-04-07 01:54:51,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a7cba3789ed4d0a8ccc7ae33f2509c7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7 2023-04-07 01:54:51,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7, entries=150, sequenceid=140, filesize=11.8 K 2023-04-07 01:54:51,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/1e64dc83b95148748d40c6d7e9bfbf00 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00 2023-04-07 01:54:51,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00, entries=150, sequenceid=140, filesize=11.8 K 2023-04-07 01:54:51,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ff332f26550df115caee283ff76382bb in 859ms, sequenceid=140, compaction requested=true 2023-04-07 01:54:51,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:51,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:51,487 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:54:51,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:51,487 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:54:51,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:51,494 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73856 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:54:51,494 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:54:51,495 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:51,495 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:51,495 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,495 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,495 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=72.1 K 2023-04-07 01:54:51,495 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=69.8 K 2023-04-07 01:54:51,495 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 98b7b77f0a6848baa5282f94c0c6f45f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832487385 2023-04-07 01:54:51,496 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bd5fcbf1f8654e82bced1edf56aa95fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832487385 2023-04-07 01:54:51,496 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9c10a300f3394e77baf59b819c358f59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680832487420 2023-04-07 01:54:51,496 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 318ffe69509741259d4e47bceefef4cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680832487420 2023-04-07 01:54:51,496 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2bd6f7d70eb743c280962eb3b298f4f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680832488043 2023-04-07 01:54:51,497 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 99c257c5e7944a4cab687c7e7917cbca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680832488645 2023-04-07 01:54:51,497 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d90aa4d8bb684c01a7a2b7381f53dff3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680832488043 2023-04-07 01:54:51,497 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 894f693ded1b4774a450be8997edbd77, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680832489179 2023-04-07 01:54:51,497 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 890ce935ca0c46338d1eb81f8862c10d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680832488645 2023-04-07 01:54:51,498 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 817ce063d5374b61a2ea347cc1403e46, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490336 2023-04-07 01:54:51,498 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e0dc28e42fcf448abf7000d0cdf3823c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680832489179 2023-04-07 01:54:51,499 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0a7cba3789ed4d0a8ccc7ae33f2509c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490339 2023-04-07 01:54:51,511 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#225 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:51,518 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#226 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:51,560 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:51,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:51,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:51,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:51,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:51,564 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:51,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:51,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:51,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:51,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:51,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:51,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:51,592 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,592 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:51,593 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:51,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:51,623 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/dc1ef85b4316408683e17f2ff9b2ebc0 2023-04-07 01:54:51,639 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/93df8997845041819b5b885c89ec7895 2023-04-07 01:54:51,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:51,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:51,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832491769, max timestamp=1680832491808Num uniques 859; 2023-04-07 01:54:51,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:51,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832491769, max timestamp=1680832491808Num uniques 859; 2023-04-07 01:54:51,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832491769, max timestamp=1680832491808Num uniques 859; 2023-04-07 01:54:51,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:51,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:51,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832551838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832551841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,924 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/1a6cfb3c84c64444bf3bca3b65c0b564 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1a6cfb3c84c64444bf3bca3b65c0b564 2023-04-07 01:54:51,929 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/38599f3d8c104d9287e82835eea566f9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/38599f3d8c104d9287e82835eea566f9 2023-04-07 01:54:51,930 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into 1a6cfb3c84c64444bf3bca3b65c0b564(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:51,930 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:51,930 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=10, startTime=1680832491487; duration=0sec 2023-04-07 01:54:51,930 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:51,930 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:54:51,933 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:54:51,933 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:51,933 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:51,933 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=69.8 K 2023-04-07 01:54:51,934 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 37ad0f9019b4426fa7647d8dd3aa4309, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680832487385 2023-04-07 01:54:51,934 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 26652907d8c4435598979dc58d82caf8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680832487420 2023-04-07 01:54:51,935 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 964cc1178399463fb88c98152e38da33, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680832488043 2023-04-07 01:54:51,935 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 411bc8541bdd4330b6c97a747dad9420, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680832488645 2023-04-07 01:54:51,936 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 497e46c1ccd641259b1187d2bd72e8c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680832489179 2023-04-07 01:54:51,936 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1e64dc83b95148748d40c6d7e9bfbf00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490339 2023-04-07 01:54:51,942 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 38599f3d8c104d9287e82835eea566f9(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:51,942 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:51,942 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=10, startTime=1680832491487; duration=0sec 2023-04-07 01:54:51,942 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:51,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832551943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:51,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832551945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:51,991 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#230 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:52,029 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/94f46c13302744549f34ef44d65acd56 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94f46c13302744549f34ef44d65acd56 2023-04-07 01:54:52,035 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into 94f46c13302744549f34ef44d65acd56(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:52,035 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,035 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=10, startTime=1680832491487; duration=0sec 2023-04-07 01:54:52,035 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:52,049 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/e92359741030405db4de45b27981a8f1 2023-04-07 01:54:52,054 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/dc1ef85b4316408683e17f2ff9b2ebc0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0 2023-04-07 01:54:52,060 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0, entries=150, sequenceid=150, filesize=11.8 K 2023-04-07 01:54:52,073 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/93df8997845041819b5b885c89ec7895 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895 2023-04-07 01:54:52,077 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895, entries=150, sequenceid=150, filesize=11.8 K 2023-04-07 01:54:52,083 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/e92359741030405db4de45b27981a8f1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1 2023-04-07 01:54:52,088 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1, entries=150, sequenceid=150, filesize=11.8 K 2023-04-07 01:54:52,088 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ff332f26550df115caee283ff76382bb in 495ms, sequenceid=150, compaction requested=false 2023-04-07 01:54:52,089 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,089 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff20c03[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,097 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,101 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,102 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,108 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,108 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:52,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832491833, max timestamp=1680832492149Num uniques 1018; 2023-04-07 01:54:52,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832491833, max timestamp=1680832492149Num uniques 1018; 2023-04-07 01:54:52,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:52,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:52,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:52,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=64.85 KB heapSize=178.80 KB 2023-04-07 01:54:52,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832491833, max timestamp=1680832492149Num uniques 1018; 2023-04-07 01:54:52,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/3768bc73e0054b0e8db972bf041d0cb2 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832492195, max timestamp=1680832492206Num uniques 859; 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832492195, max timestamp=1680832492206Num uniques 859; 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832492195, max timestamp=1680832492206Num uniques 859; 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:52,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:54:52,208 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/f97386d4d8d140adac7ef3a140efe3e9 2023-04-07 01:54:52,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,216 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,224 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,230 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,230 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,230 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:52,230 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/76ee3b18012c4f67a35565fbe8be3aa5 2023-04-07 01:54:52,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28de0c45[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/3768bc73e0054b0e8db972bf041d0cb2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2 2023-04-07 01:54:52,237 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2, entries=200, sequenceid=192, filesize=14.1 K 2023-04-07 01:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/f97386d4d8d140adac7ef3a140efe3e9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9 2023-04-07 01:54:52,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,252 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,252 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832492221, max timestamp=1680832492248Num uniques 1068; 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832492221, max timestamp=1680832492248Num uniques 1068; 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832492221, max timestamp=1680832492248Num uniques 1068; 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 01:54:52,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 01:54:52,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,254 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,254 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9, entries=150, sequenceid=192, filesize=11.8 K 2023-04-07 01:54:52,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/76ee3b18012c4f67a35565fbe8be3aa5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5 2023-04-07 01:54:52,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5, entries=150, sequenceid=192, filesize=11.8 K 2023-04-07 01:54:52,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=87.22 KB/89310 for ff332f26550df115caee283ff76382bb in 117ms, sequenceid=192, compaction requested=true 2023-04-07 01:54:52,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:52,272 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:52,272 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:52,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:52,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:52,273 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:52,273 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:52,273 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:52,273 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,273 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:52,273 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1a6cfb3c84c64444bf3bca3b65c0b564, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=37.8 K 2023-04-07 01:54:52,273 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,273 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/38599f3d8c104d9287e82835eea566f9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=35.5 K 2023-04-07 01:54:52,273 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1a6cfb3c84c64444bf3bca3b65c0b564, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490339 2023-04-07 01:54:52,274 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 38599f3d8c104d9287e82835eea566f9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490339 2023-04-07 01:54:52,274 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 93df8997845041819b5b885c89ec7895, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680832490643 2023-04-07 01:54:52,274 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting dc1ef85b4316408683e17f2ff9b2ebc0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680832490643 2023-04-07 01:54:52,274 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f97386d4d8d140adac7ef3a140efe3e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491840 2023-04-07 01:54:52,274 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3768bc73e0054b0e8db972bf041d0cb2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491837 2023-04-07 01:54:52,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:52,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=93.93 KB heapSize=264.61 KB 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:52,282 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:52,286 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#236 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:52,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/86a844de2bb0486eb9bd06fcebb582f2 2023-04-07 01:54:52,303 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/95074500653d4f4180e9f9a80f350295 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/95074500653d4f4180e9f9a80f350295 2023-04-07 01:54:52,308 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/1fed4a21871a45fb9c05baf9cb101094 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1fed4a21871a45fb9c05baf9cb101094 2023-04-07 01:54:52,310 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 95074500653d4f4180e9f9a80f350295(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:52,310 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,310 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=13, startTime=1680832492272; duration=0sec 2023-04-07 01:54:52,310 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:52,310 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:52,311 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:52,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:52,312 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,312 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94f46c13302744549f34ef44d65acd56, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=35.5 K 2023-04-07 01:54:52,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 94f46c13302744549f34ef44d65acd56, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680832490339 2023-04-07 01:54:52,312 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into 1fed4a21871a45fb9c05baf9cb101094(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:52,312 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,312 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=13, startTime=1680832492272; duration=0sec 2023-04-07 01:54:52,313 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:52,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,315 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e92359741030405db4de45b27981a8f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680832490643 2023-04-07 01:54:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832552308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,315 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 76ee3b18012c4f67a35565fbe8be3aa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491840 2023-04-07 01:54:52,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832552312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,323 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:52,347 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/c40efa72c4a1479a90293113c4edfe86 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c40efa72c4a1479a90293113c4edfe86 2023-04-07 01:54:52,354 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,355 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into c40efa72c4a1479a90293113c4edfe86(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:52,356 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,356 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=13, startTime=1680832492272; duration=0sec 2023-04-07 01:54:52,356 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:52,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,372 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,374 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,375 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,375 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,376 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,377 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,377 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:52,377 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,378 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,379 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,380 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,380 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e976464[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,384 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,388 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,391 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,391 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,397 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:52,398 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832552416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832552420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832552461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832552467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,498 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,501 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,503 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,503 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,504 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,504 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,505 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:52,505 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,507 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564ce6a4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,510 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,512 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,514 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,514 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,517 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,521 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,522 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,522 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,522 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,522 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,522 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832552618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,622 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832552623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,626 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,628 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,628 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,631 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:52,632 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a17a7a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,639 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,644 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,644 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,646 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,646 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,646 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,646 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,647 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,650 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,650 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/6d039e8c1d1f4f079886fb2c1c7e8f8a 2023-04-07 01:54:52,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/a8ef250e2a544dff9c63ddde9d78aaac 2023-04-07 01:54:52,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/86a844de2bb0486eb9bd06fcebb582f2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2 2023-04-07 01:54:52,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2, entries=200, sequenceid=241, filesize=14.1 K 2023-04-07 01:54:52,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/6d039e8c1d1f4f079886fb2c1c7e8f8a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a 2023-04-07 01:54:52,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a, entries=150, sequenceid=241, filesize=11.8 K 2023-04-07 01:54:52,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/a8ef250e2a544dff9c63ddde9d78aaac as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac 2023-04-07 01:54:52,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac, entries=150, sequenceid=241, filesize=11.8 K 2023-04-07 01:54:52,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~284.48 KB/291312, currentSize=80.51 KB/82440 for ff332f26550df115caee283ff76382bb in 469ms, sequenceid=241, compaction requested=false 2023-04-07 01:54:52,745 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,750 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,753 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,753 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,759 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,760 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,761 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:52,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,770 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/446292a2e61041128a8812dcff01823f 2023-04-07 01:54:52,779 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/15b0d202045744aa9144059dad2ee422 2023-04-07 01:54:52,788 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/c35e5f30f4a24e71becf6d3008333f47 2023-04-07 01:54:52,792 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/446292a2e61041128a8812dcff01823f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f 2023-04-07 01:54:52,795 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f, entries=150, sequenceid=259, filesize=11.8 K 2023-04-07 01:54:52,796 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/15b0d202045744aa9144059dad2ee422 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422 2023-04-07 01:54:52,801 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422, entries=150, sequenceid=259, filesize=11.8 K 2023-04-07 01:54:52,802 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/c35e5f30f4a24e71becf6d3008333f47 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47 2023-04-07 01:54:52,809 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47, entries=150, sequenceid=259, filesize=11.8 K 2023-04-07 01:54:52,810 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for ff332f26550df115caee283ff76382bb in 49ms, sequenceid=259, compaction requested=true 2023-04-07 01:54:52,810 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:52,810 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c5f72f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,827 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,832 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,832 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,836 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,836 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,840 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:52,846 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,847 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,847 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832492858, max timestamp=1680832492885Num uniques 859; 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832492858, max timestamp=1680832492885Num uniques 859; 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832492858, max timestamp=1680832492885Num uniques 859; 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:52,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832492898, max timestamp=1680832492927Num uniques 1018; 2023-04-07 01:54:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832492898, max timestamp=1680832492927Num uniques 1018; 2023-04-07 01:54:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:52,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832492898, max timestamp=1680832492927Num uniques 1018; 2023-04-07 01:54:52,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:54:52,947 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:52,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:52,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:52,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:52,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:52,950 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:52,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:52,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:52,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832552947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832552948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,952 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,952 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,952 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:52,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:52,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:52,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832552950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:52,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:52,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:52,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:52,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:52,956 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f40937[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:52,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:52,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,960 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,960 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:52,960 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:52,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:52,960 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:52,965 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:52,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:52,967 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:52,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:52,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:52,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:52,967 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:52,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:52,969 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:52,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,974 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:52,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:52,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:52,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:52,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:52,975 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:52,975 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:52,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:52,975 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:52,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832553052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832553052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832553054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,075 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,078 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,078 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,078 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,078 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,079 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,079 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,082 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,082 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,082 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:53,082 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,084 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453ca8cb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,086 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,086 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,087 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,092 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,092 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,092 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,092 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,092 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,095 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,098 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,098 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,199 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,202 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,205 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,212 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ec78bc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,220 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,220 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,220 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,220 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,220 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,228 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,228 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,228 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,228 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,228 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,229 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,229 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,229 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,229 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/e394eb157fea45c993a6bb41595d0d67 2023-04-07 01:54:53,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832553255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60746 deadline: 1680832553256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832553259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,272 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/b2ebed7388ce403094d06df8f14ad80f 2023-04-07 01:54:53,329 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,334 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,335 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,337 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,337 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,338 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:53,338 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,339 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ce97e1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,343 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,349 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,349 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,350 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,350 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,350 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:53,357 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,358 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,458 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,461 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,461 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,461 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,461 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,461 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,461 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,463 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,465 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,465 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,465 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:53,465 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,466 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,466 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d278c5e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,469 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,471 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,473 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,473 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,476 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,476 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,476 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,479 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:53,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,480 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,480 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60708 deadline: 1680832553560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832553562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:53,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1680832553564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,580 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,584 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,586 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,587 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,588 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,588 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:53,588 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,589 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dfd1bc1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,599 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,599 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,602 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,602 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,602 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,603 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,608 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,608 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/1256421390554f418c4bd672a5a9fbaf 2023-04-07 01:54:53,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/e394eb157fea45c993a6bb41595d0d67 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67 2023-04-07 01:54:53,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67, entries=100, sequenceid=269, filesize=9.5 K 2023-04-07 01:54:53,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/b2ebed7388ce403094d06df8f14ad80f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f 2023-04-07 01:54:53,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f, entries=100, sequenceid=269, filesize=9.5 K 2023-04-07 01:54:53,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/1256421390554f418c4bd672a5a9fbaf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf 2023-04-07 01:54:53,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf, entries=100, sequenceid=269, filesize=9.5 K 2023-04-07 01:54:53,708 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ff332f26550df115caee283ff76382bb in 876ms, sequenceid=269, compaction requested=true 2023-04-07 01:54:53,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:53,710 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:53,710 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:53,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,716 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,716 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,716 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46158 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:53,716 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:53,716 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,716 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/95074500653d4f4180e9f9a80f350295, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=45.1 K 2023-04-07 01:54:53,717 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48548 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:53,717 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:53,717 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,717 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 95074500653d4f4180e9f9a80f350295, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491840 2023-04-07 01:54:53,717 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1fed4a21871a45fb9c05baf9cb101094, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=47.4 K 2023-04-07 01:54:53,717 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6d039e8c1d1f4f079886fb2c1c7e8f8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680832492259 2023-04-07 01:54:53,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,718 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,719 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,719 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,719 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:53,720 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1fed4a21871a45fb9c05baf9cb101094, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491840 2023-04-07 01:54:53,720 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 15b0d202045744aa9144059dad2ee422, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680832492303 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 86a844de2bb0486eb9bd06fcebb582f2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680832492259 2023-04-07 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b2ebed7388ce403094d06df8f14ad80f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680832492830 2023-04-07 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 446292a2e61041128a8812dcff01823f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680832492303 2023-04-07 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e394eb157fea45c993a6bb41595d0d67, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680832492830 2023-04-07 01:54:53,732 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#247 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:53,735 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#248 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:53,738 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/fab4d7929f364bccab0957a21271fe05 2023-04-07 01:54:53,757 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/6b413e9eafcd4f5997258855b8804314 2023-04-07 01:54:53,767 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/23b0ec4919df4b3b97db7915c9a6c4fc 2023-04-07 01:54:53,770 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/fab4d7929f364bccab0957a21271fe05 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05 2023-04-07 01:54:53,774 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05, entries=150, sequenceid=322, filesize=11.9 K 2023-04-07 01:54:53,775 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/6b413e9eafcd4f5997258855b8804314 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314 2023-04-07 01:54:53,803 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314, entries=150, sequenceid=322, filesize=11.9 K 2023-04-07 01:54:53,804 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/23b0ec4919df4b3b97db7915c9a6c4fc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc 2023-04-07 01:54:53,812 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc, entries=150, sequenceid=322, filesize=11.9 K 2023-04-07 01:54:53,813 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=0 B/0 for ff332f26550df115caee283ff76382bb in 94ms, sequenceid=322, compaction requested=true 2023-04-07 01:54:53,813 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:53,813 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4950a35a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,824 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,824 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,836 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,836 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:53,936 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:53,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:53,939 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:53,939 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,939 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:53,939 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:53,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,941 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:53,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:53,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:53,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:53,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:53,945 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec540e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:53,947 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:53,947 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:53,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:53,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:53,953 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:53,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:53,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:53,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:53,953 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:53,955 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,955 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,955 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:53,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:53,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:53,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:53,964 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:53,965 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:53,965 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:53,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:53,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:54,065 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:54,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:54,069 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:54,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:54,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:54,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:54,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:54,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:54,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:54,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:54,073 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 01:54:54,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:54,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:54,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:54,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:54,075 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:54,075 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:54,075 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:54,075 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:54,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832494092, max timestamp=1680832494099Num uniques 859; 2023-04-07 01:54:54,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:54,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832494092, max timestamp=1680832494099Num uniques 859; 2023-04-07 01:54:54,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832494092, max timestamp=1680832494099Num uniques 859; 2023-04-07 01:54:54,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:54,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:54:54,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832554123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832554123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832554123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,151 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/a55a427388574f338c0dc582570112d6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a55a427388574f338c0dc582570112d6 2023-04-07 01:54:54,156 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/c7a0e7aa862446a0b737d553935e3d55 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7a0e7aa862446a0b737d553935e3d55 2023-04-07 01:54:54,158 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into a55a427388574f338c0dc582570112d6(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:54,158 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:54,158 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=12, startTime=1680832493710; duration=0sec 2023-04-07 01:54:54,158 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:54,158 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:54,160 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58344 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:54,160 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:54,160 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:54,161 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c40efa72c4a1479a90293113c4edfe86, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=57.0 K 2023-04-07 01:54:54,161 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c40efa72c4a1479a90293113c4edfe86, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680832491840 2023-04-07 01:54:54,161 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a8ef250e2a544dff9c63ddde9d78aaac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680832492259 2023-04-07 01:54:54,162 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c35e5f30f4a24e71becf6d3008333f47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1680832492303 2023-04-07 01:54:54,162 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1256421390554f418c4bd672a5a9fbaf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680832492830 2023-04-07 01:54:54,163 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 23b0ec4919df4b3b97db7915c9a6c4fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680832492936 2023-04-07 01:54:54,163 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into c7a0e7aa862446a0b737d553935e3d55(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:54,163 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:54,163 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=12, startTime=1680832493710; duration=0sec 2023-04-07 01:54:54,164 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:54,172 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832554228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832554228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832554228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832554433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832554434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832554436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832554468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,471 DEBUG [Thread-1666] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:54,486 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/53386f21806841f681876097bb4d78e6 2023-04-07 01:54:54,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832554486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,490 DEBUG [Thread-1672] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:54,614 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/cc49b46d1663475da1c3e8185b3b49cc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/cc49b46d1663475da1c3e8185b3b49cc 2023-04-07 01:54:54,618 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into cc49b46d1663475da1c3e8185b3b49cc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:54,618 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:54,618 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=11, startTime=1680832493710; duration=0sec 2023-04-07 01:54:54,618 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:54,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832554739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832554739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832554740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,898 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a643051482847cb8830a5fe59eb6362 2023-04-07 01:54:54,914 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/0ec103e37c044791af5b6d5ee318b155 2023-04-07 01:54:54,920 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/53386f21806841f681876097bb4d78e6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6 2023-04-07 01:54:54,924 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6, entries=200, sequenceid=333, filesize=14.3 K 2023-04-07 01:54:54,925 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a643051482847cb8830a5fe59eb6362 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362 2023-04-07 01:54:54,929 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362, entries=150, sequenceid=333, filesize=11.9 K 2023-04-07 01:54:54,930 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/0ec103e37c044791af5b6d5ee318b155 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155 2023-04-07 01:54:54,938 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155, entries=150, sequenceid=333, filesize=11.9 K 2023-04-07 01:54:54,939 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for ff332f26550df115caee283ff76382bb in 866ms, sequenceid=333, compaction requested=true 2023-04-07 01:54:54,939 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:54,939 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:54,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:54,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:54,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:54,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55786db1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:54,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:54,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,947 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:54,950 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:54,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:54,950 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:54,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:54,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:54,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:54,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:54,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:54,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:54,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:54,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:54,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:54,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:54,959 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:54,959 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,067 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:55,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:55,070 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:55,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:55,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:55,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:55,074 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:55,074 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,087 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/a8b87695bd8f4434bfb4ef19a7961085 2023-04-07 01:54:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:55,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832555249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832555249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832555252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832555353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832555353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832555356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,497 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/8e94c5240aaa4f719eb9355f13ecfed4 2023-04-07 01:54:55,534 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/62b8050c4ea447e8b1a77c301006956d 2023-04-07 01:54:55,540 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/a8b87695bd8f4434bfb4ef19a7961085 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085 2023-04-07 01:54:55,547 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085, entries=150, sequenceid=372, filesize=11.9 K 2023-04-07 01:54:55,549 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/8e94c5240aaa4f719eb9355f13ecfed4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4 2023-04-07 01:54:55,553 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4, entries=150, sequenceid=372, filesize=11.9 K 2023-04-07 01:54:55,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/62b8050c4ea447e8b1a77c301006956d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d 2023-04-07 01:54:55,558 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d, entries=150, sequenceid=372, filesize=11.9 K 2023-04-07 01:54:55,558 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for ff332f26550df115caee283ff76382bb in 484ms, sequenceid=372, compaction requested=true 2023-04-07 01:54:55,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:55,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:55,561 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe47cc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:55,565 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:55,565 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:55,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:55,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:55,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:55,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:55,573 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:55,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:55,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:55,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:55,573 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:55,576 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,576 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,576 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,580 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:55,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,585 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:55,585 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:55,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:55,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/5ea576ea2dcb4d7da1c3b1961b39a305 2023-04-07 01:54:55,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832555599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832555602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832555602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/da36948101bd48449313f4a3f6031bbd 2023-04-07 01:54:55,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/892ce032143f47a2bf1d6e24cca4a7a7 2023-04-07 01:54:55,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/5ea576ea2dcb4d7da1c3b1961b39a305 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305 2023-04-07 01:54:55,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305, entries=200, sequenceid=384, filesize=14.3 K 2023-04-07 01:54:55,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/da36948101bd48449313f4a3f6031bbd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd 2023-04-07 01:54:55,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd, entries=150, sequenceid=384, filesize=11.9 K 2023-04-07 01:54:55,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/892ce032143f47a2bf1d6e24cca4a7a7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7 2023-04-07 01:54:55,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7, entries=150, sequenceid=384, filesize=11.9 K 2023-04-07 01:54:55,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ff332f26550df115caee283ff76382bb in 102ms, sequenceid=384, compaction requested=true 2023-04-07 01:54:55,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:55,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:55,669 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:55,669 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:55,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:55,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:55,671 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66204 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:55,671 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:55,671 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61324 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:55,671 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,671 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:55,671 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7a0e7aa862446a0b737d553935e3d55, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=64.7 K 2023-04-07 01:54:55,671 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,671 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a55a427388574f338c0dc582570112d6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=59.9 K 2023-04-07 01:54:55,672 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c7a0e7aa862446a0b737d553935e3d55, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680832492303 2023-04-07 01:54:55,672 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a55a427388574f338c0dc582570112d6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680832492303 2023-04-07 01:54:55,672 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting fab4d7929f364bccab0957a21271fe05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680832492936 2023-04-07 01:54:55,672 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6b413e9eafcd4f5997258855b8804314, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680832492936 2023-04-07 01:54:55,672 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 53386f21806841f681876097bb4d78e6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680832494070 2023-04-07 01:54:55,680 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0a643051482847cb8830a5fe59eb6362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680832494070 2023-04-07 01:54:55,680 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a8b87695bd8f4434bfb4ef19a7961085, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680832494117 2023-04-07 01:54:55,680 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8e94c5240aaa4f719eb9355f13ecfed4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680832494117 2023-04-07 01:54:55,680 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5ea576ea2dcb4d7da1c3b1961b39a305, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:55,681 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting da36948101bd48449313f4a3f6031bbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:55,685 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:55,696 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:55,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:55,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:55,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,697 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:55,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:55,707 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#261 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:55,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:55,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:55,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:55,720 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#262 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:55,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832555722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832555725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832555727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf870fa[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:55,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:55,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:55,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:55,764 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:55,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:55,764 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:55,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,767 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,772 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:55,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,772 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:55,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,772 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:55,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,815 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/7c021b12a7e74edd8b8da032a4b5ba03 2023-04-07 01:54:55,819 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/08808cfc51b542079f37c7080455d562 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/08808cfc51b542079f37c7080455d562 2023-04-07 01:54:55,827 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 08808cfc51b542079f37c7080455d562(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:55,827 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:55,827 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=11, startTime=1680832495669; duration=0sec 2023-04-07 01:54:55,827 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:55,827 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832555828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832555828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,830 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:55,830 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:55,830 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,830 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/cc49b46d1663475da1c3e8185b3b49cc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.1 K 2023-04-07 01:54:55,831 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting cc49b46d1663475da1c3e8185b3b49cc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680832492936 2023-04-07 01:54:55,831 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ede9ecc6f92b4b4d816e2eeb79dc2059 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ede9ecc6f92b4b4d816e2eeb79dc2059 2023-04-07 01:54:55,831 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0ec103e37c044791af5b6d5ee318b155, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680832494070 2023-04-07 01:54:55,832 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 62b8050c4ea447e8b1a77c301006956d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680832494117 2023-04-07 01:54:55,832 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 892ce032143f47a2bf1d6e24cca4a7a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:55,836 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into ede9ecc6f92b4b4d816e2eeb79dc2059(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:55,836 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:55,836 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=11, startTime=1680832495669; duration=0sec 2023-04-07 01:54:55,836 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:55,842 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:55,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:55,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832555832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,862 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/6e3f74917b0c4340ae4601bd69f76f0a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6e3f74917b0c4340ae4601bd69f76f0a 2023-04-07 01:54:55,866 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into 6e3f74917b0c4340ae4601bd69f76f0a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:55,866 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:55,866 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=12, startTime=1680832495670; duration=0sec 2023-04-07 01:54:55,867 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:55,873 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:55,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:55,877 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:55,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:55,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:55,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,879 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:55,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:55,882 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,882 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:55,882 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:55,882 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:55,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad21f4f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:55,887 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:55,887 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,889 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:55,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:55,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,892 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:55,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:55,902 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:55,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:55,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:55,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:55,902 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:55,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,906 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:55,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:55,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:55,931 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:55,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:55,931 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:55,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:55,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,031 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832556030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832556031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,039 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,042 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832556044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,051 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,068 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1e441[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,076 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,079 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,079 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,079 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,081 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,087 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,087 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,091 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,093 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,096 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,096 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,096 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,097 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,097 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,097 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,097 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,097 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,097 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,197 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,201 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,202 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,204 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,210 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e930a30[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,212 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,212 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,214 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,214 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,222 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,224 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:56,225 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,225 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/d462679421184deba725a344acbec530 2023-04-07 01:54:56,325 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,328 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,332 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,332 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,332 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,332 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,335 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832556335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832556335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3113ccb8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,342 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,342 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,344 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,344 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,344 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,348 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832556349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,361 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:56,362 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,362 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,462 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,465 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,465 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,467 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,468 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,469 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,469 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf185b2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,474 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,477 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,477 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,477 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,480 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,480 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,483 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,487 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,488 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,488 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,588 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,591 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,593 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,593 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,593 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,593 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,594 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,598 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c18561[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,600 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,600 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,606 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,608 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,608 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,611 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,611 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,611 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,617 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,617 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/780446d018d64502a46720af8f7da8d9 2023-04-07 01:54:56,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/7c021b12a7e74edd8b8da032a4b5ba03 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03 2023-04-07 01:54:56,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03, entries=200, sequenceid=405, filesize=14.3 K 2023-04-07 01:54:56,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/d462679421184deba725a344acbec530 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530 2023-04-07 01:54:56,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530, entries=150, sequenceid=405, filesize=11.9 K 2023-04-07 01:54:56,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/780446d018d64502a46720af8f7da8d9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9 2023-04-07 01:54:56,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9, entries=150, sequenceid=405, filesize=11.9 K 2023-04-07 01:54:56,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ff332f26550df115caee283ff76382bb in 993ms, sequenceid=405, compaction requested=false 2023-04-07 01:54:56,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:56,717 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:56,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:56,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:56,728 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:56,728 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,728 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:56,728 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:56,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,731 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:56,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:56,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:56,754 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:56,754 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:56,797 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/312b6bbba68744a8a87d50306d3ba029 2023-04-07 01:54:56,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:56,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:56,847 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0495807ea70a4a479e426b4bfaefb8a1 2023-04-07 01:54:56,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832496856, max timestamp=1680832496861Num uniques 859; 2023-04-07 01:54:56,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:56,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832556870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832556871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:56,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832556872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,894 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/964c6b7de04e402b8128db0a0db36295 2023-04-07 01:54:56,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/312b6bbba68744a8a87d50306d3ba029 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029 2023-04-07 01:54:56,904 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029, entries=100, sequenceid=419, filesize=9.5 K 2023-04-07 01:54:56,905 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0495807ea70a4a479e426b4bfaefb8a1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1 2023-04-07 01:54:56,910 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1, entries=100, sequenceid=419, filesize=9.5 K 2023-04-07 01:54:56,911 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/964c6b7de04e402b8128db0a0db36295 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295 2023-04-07 01:54:56,915 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295, entries=100, sequenceid=419, filesize=9.5 K 2023-04-07 01:54:56,916 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for ff332f26550df115caee283ff76382bb in 163ms, sequenceid=419, compaction requested=true 2023-04-07 01:54:56,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:56,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:56,918 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,919 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,919 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e55cac3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:56,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:56,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,924 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:56,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:56,927 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:56,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:56,932 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:56,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:56,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:56,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:56,932 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:56,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:56,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,936 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,936 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:56,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:56,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:56,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:56,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:56,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:56,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:56,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:56,943 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:56,943 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:56,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:56,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=129.71 KB heapSize=381.64 KB 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:56,992 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/15689a4d8f2f455fb27203dcb0fb8730 2023-04-07 01:54:57,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/09b3ce5ed80b4d17a90034716836d217 2023-04-07 01:54:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832557027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832557027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832557030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,043 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,047 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:57,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418cbdbd[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,060 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,070 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,072 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,079 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,080 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,080 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,079 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,080 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832557131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832557131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832557134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,180 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,193 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,199 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,199 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,199 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:57,199 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,201 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36276c2b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,205 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:57,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,210 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,210 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,222 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,222 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,224 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,224 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,324 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,329 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:57,334 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,336 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,336 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832557335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832557335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832557338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f91d03[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,341 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,350 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,350 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:57,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,360 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,360 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,360 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,362 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,362 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,362 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,363 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,365 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,369 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,369 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,369 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/94a60e63dd1d40d2a2f6456e2a9e3302 2023-04-07 01:54:57,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/15689a4d8f2f455fb27203dcb0fb8730 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730 2023-04-07 01:54:57,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730, entries=200, sequenceid=446, filesize=14.3 K 2023-04-07 01:54:57,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/09b3ce5ed80b4d17a90034716836d217 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217 2023-04-07 01:54:57,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217, entries=150, sequenceid=446, filesize=11.9 K 2023-04-07 01:54:57,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/94a60e63dd1d40d2a2f6456e2a9e3302 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302 2023-04-07 01:54:57,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302, entries=150, sequenceid=446, filesize=11.9 K 2023-04-07 01:54:57,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~381.59 KB/390752, currentSize=46.96 KB/48090 for ff332f26550df115caee283ff76382bb in 455ms, sequenceid=446, compaction requested=true 2023-04-07 01:54:57,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:57,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:57,432 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:57,432 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:57,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:57,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:57,434 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51794 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:57,434 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:57,434 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,434 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ede9ecc6f92b4b4d816e2eeb79dc2059, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=50.6 K 2023-04-07 01:54:57,434 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:57,434 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:57,435 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ede9ecc6f92b4b4d816e2eeb79dc2059, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:57,435 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,435 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/08808cfc51b542079f37c7080455d562, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=45.8 K 2023-04-07 01:54:57,435 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7c021b12a7e74edd8b8da032a4b5ba03, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680832495578 2023-04-07 01:54:57,435 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 08808cfc51b542079f37c7080455d562, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:57,436 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 312b6bbba68744a8a87d50306d3ba029, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680832495720 2023-04-07 01:54:57,436 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d462679421184deba725a344acbec530, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680832495578 2023-04-07 01:54:57,436 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 15689a4d8f2f455fb27203dcb0fb8730, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680832496862 2023-04-07 01:54:57,437 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0495807ea70a4a479e426b4bfaefb8a1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680832495720 2023-04-07 01:54:57,437 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 09b3ce5ed80b4d17a90034716836d217, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680832496862 2023-04-07 01:54:57,448 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:57,450 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:57,470 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,488 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,489 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,490 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,512 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,512 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,512 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:54:57,513 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:57,514 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,514 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:57,514 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,514 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:57,514 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,531 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ef2c2c02de704fb78bba0c86233d9e0f 2023-04-07 01:54:57,554 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/fa6886591bfd4e3dbff40f00044c7882 2023-04-07 01:54:57,586 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/d510368af35c49f5bdb578c99513f68b 2023-04-07 01:54:57,591 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ef2c2c02de704fb78bba0c86233d9e0f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f 2023-04-07 01:54:57,596 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f, entries=150, sequenceid=456, filesize=11.9 K 2023-04-07 01:54:57,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/fa6886591bfd4e3dbff40f00044c7882 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882 2023-04-07 01:54:57,601 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882, entries=150, sequenceid=456, filesize=11.9 K 2023-04-07 01:54:57,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/d510368af35c49f5bdb578c99513f68b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b 2023-04-07 01:54:57,626 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b, entries=150, sequenceid=456, filesize=11.9 K 2023-04-07 01:54:57,627 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for ff332f26550df115caee283ff76382bb in 115ms, sequenceid=456, compaction requested=true 2023-04-07 01:54:57,627 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:57,627 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,631 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423adc4d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,634 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,634 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,635 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,638 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,638 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,640 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,640 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,654 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,654 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:57,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:57,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832557732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832497697, max timestamp=1680832497735Num uniques 859; 2023-04-07 01:54:57,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:57,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832497697, max timestamp=1680832497735Num uniques 859; 2023-04-07 01:54:57,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:57,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832497697, max timestamp=1680832497735Num uniques 859; 2023-04-07 01:54:57,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:54:57,754 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,757 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,757 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,758 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:57,761 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ffe0e2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,770 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,772 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,773 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832557769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832557770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,776 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,785 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,785 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,785 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832557837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832557874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:57,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832557874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,885 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:57,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:57,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:57,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:57,889 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:57,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:57,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:57,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,889 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,896 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:57,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:57,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:57,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:57,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:57,901 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b348e22[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:57,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:57,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,905 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ce0950768318401d89b96f6c28a4bdcf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ce0950768318401d89b96f6c28a4bdcf 2023-04-07 01:54:57,906 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,906 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:57,906 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:57,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:57,912 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,912 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into ce0950768318401d89b96f6c28a4bdcf(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:57,912 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:57,912 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=12, startTime=1680832497432; duration=0sec 2023-04-07 01:54:57,912 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:57,912 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:54:57,914 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:54:57,914 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:57,914 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:57,914 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6e3f74917b0c4340ae4601bd69f76f0a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=57.7 K 2023-04-07 01:54:57,915 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6e3f74917b0c4340ae4601bd69f76f0a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680832495252 2023-04-07 01:54:57,915 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 780446d018d64502a46720af8f7da8d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680832495578 2023-04-07 01:54:57,916 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 964c6b7de04e402b8128db0a0db36295, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680832495720 2023-04-07 01:54:57,916 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 94a60e63dd1d40d2a2f6456e2a9e3302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680832496862 2023-04-07 01:54:57,917 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d510368af35c49f5bdb578c99513f68b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680832496979 2023-04-07 01:54:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:57,915 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/5321e3fbc12c4daab65167ff1afa54de as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5321e3fbc12c4daab65167ff1afa54de 2023-04-07 01:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:57,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:57,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:57,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:57,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:57,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:57,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:57,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:57,928 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 5321e3fbc12c4daab65167ff1afa54de(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:57,928 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:57,928 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=12, startTime=1680832497432; duration=0sec 2023-04-07 01:54:57,928 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:57,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:57,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:57,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:57,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:57,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:57,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:57,932 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:57,933 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:57,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:57,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:57,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:57,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:57,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:57,946 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#279 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:57,960 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/b940a79086da46db8d9cb212c860bc3f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b940a79086da46db8d9cb212c860bc3f 2023-04-07 01:54:57,965 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into b940a79086da46db8d9cb212c860bc3f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:57,966 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:57,966 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=11, startTime=1680832497432; duration=0sec 2023-04-07 01:54:57,966 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:58,033 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:58,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:58,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:58,036 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:58,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:58,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:58,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:58,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:58,044 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832558042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:58,046 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216f9742[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:58,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:58,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:58,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:58,056 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:58,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:58,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:58,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:58,056 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:58,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:58,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:58,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,064 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,064 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/b07c79c1584b4073917c76069d684e33 2023-04-07 01:54:58,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832558079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832558079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,164 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:58,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:58,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:58,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:58,167 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:58,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:58,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:58,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,170 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:58,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:58,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:58,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:58,172 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:58,174 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,174 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3523ff96[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:58,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:58,179 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:58,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:58,182 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:58,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:58,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:58,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:58,182 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:58,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:58,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:58,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,189 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:58,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:58,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,189 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:58,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,190 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,290 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:58,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:58,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:58,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:58,295 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:58,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:58,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:58,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:58,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:58,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:58,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:58,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,303 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:58,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:58,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:58,304 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:58,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:58,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc2e120[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:58,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:58,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,310 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:58,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:58,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:58,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:58,314 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:58,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:58,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:58,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:58,314 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:58,316 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,316 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,316 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:58,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:58,322 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:58,322 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:58,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832558347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832558382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832558383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,423 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:58,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:58,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:58,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:58,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:58,426 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:58,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:58,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:58,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,427 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:58,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:58,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:58,429 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,429 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:58,429 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:58,429 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:58,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63482f04[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:58,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:58,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:58,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:58,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:58,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:58,438 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:58,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:58,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:58,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:58,438 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:58,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:58,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:58,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:58,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:58,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:58,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:58,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:58,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:58,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:58,445 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:58,445 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:58,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bbae6fc220e74115a2efdcf8cb740c8a 2023-04-07 01:54:58,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60784 deadline: 1680832558499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,503 DEBUG [Thread-1672] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:58,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60768 deadline: 1680832558505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,509 DEBUG [Thread-1666] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:54:58,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/52dffc8858014b12a6521a0e4b743cae 2023-04-07 01:54:58,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/b07c79c1584b4073917c76069d684e33 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33 2023-04-07 01:54:58,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33, entries=250, sequenceid=467, filesize=16.7 K 2023-04-07 01:54:58,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bbae6fc220e74115a2efdcf8cb740c8a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a 2023-04-07 01:54:58,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a, entries=150, sequenceid=467, filesize=11.9 K 2023-04-07 01:54:58,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/52dffc8858014b12a6521a0e4b743cae as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae 2023-04-07 01:54:58,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae, entries=150, sequenceid=467, filesize=11.9 K 2023-04-07 01:54:58,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ff332f26550df115caee283ff76382bb in 882ms, sequenceid=467, compaction requested=true 2023-04-07 01:54:58,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:58,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:58,539 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:58,539 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:58,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:58,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:58,540 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42188 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:58,540 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:58,540 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:58,540 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,540 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5321e3fbc12c4daab65167ff1afa54de, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=36.4 K 2023-04-07 01:54:58,540 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:58,541 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5321e3fbc12c4daab65167ff1afa54de, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680832496862 2023-04-07 01:54:58,541 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fa6886591bfd4e3dbff40f00044c7882, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680832496979 2023-04-07 01:54:58,542 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bbae6fc220e74115a2efdcf8cb740c8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680832497648 2023-04-07 01:54:58,545 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:58,547 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:58,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:58,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:58,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:58,549 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:58,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:58,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:58,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,550 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:58,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:58,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:58,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:58,553 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,554 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ce0950768318401d89b96f6c28a4bdcf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=41.2 K 2023-04-07 01:54:58,554 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ce0950768318401d89b96f6c28a4bdcf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680832496862 2023-04-07 01:54:58,554 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ef2c2c02de704fb78bba0c86233d9e0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680832496979 2023-04-07 01:54:58,555 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b07c79c1584b4073917c76069d684e33, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680832497648 2023-04-07 01:54:58,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:58,566 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:58,569 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:58,569 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:58,569 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:58,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:54:58,573 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:58,630 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/c7d97a6d7e08437c8eec2e0dc4011fca as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7d97a6d7e08437c8eec2e0dc4011fca 2023-04-07 01:54:58,636 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into c7d97a6d7e08437c8eec2e0dc4011fca(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:58,636 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:58,636 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=13, startTime=1680832498538; duration=0sec 2023-04-07 01:54:58,637 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:58,637 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:54:58,637 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:54:58,638 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:54:58,638 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. because compaction request was cancelled 2023-04-07 01:54:58,641 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f 2023-04-07 01:54:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:58,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:58,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832558876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832558887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832558889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:58,958 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/dae637f7d95b4fc5b43ad017d08c9d2c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/dae637f7d95b4fc5b43ad017d08c9d2c 2023-04-07 01:54:58,962 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into dae637f7d95b4fc5b43ad017d08c9d2c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:58,962 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:58,962 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=13, startTime=1680832498539; duration=0sec 2023-04-07 01:54:58,962 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:58,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:58,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832558979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,070 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/30fcbd7c83cd4ed0bc411ec80f41eaca 2023-04-07 01:54:59,080 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/79cefa46c85a48549483ba028d9a774f 2023-04-07 01:54:59,083 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f 2023-04-07 01:54:59,087 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:59,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/30fcbd7c83cd4ed0bc411ec80f41eaca as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca 2023-04-07 01:54:59,092 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:59,093 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/79cefa46c85a48549483ba028d9a774f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f 2023-04-07 01:54:59,095 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f, entries=150, sequenceid=506, filesize=11.9 K 2023-04-07 01:54:59,096 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for ff332f26550df115caee283ff76382bb in 527ms, sequenceid=506, compaction requested=true 2023-04-07 01:54:59,096 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:59,096 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:59,099 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec071a8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:59,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:59,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:59,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:59,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:59,102 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,102 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:59,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,103 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,103 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,103 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:59,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:59,105 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:59,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:59,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:59,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:59,105 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:59,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,109 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,113 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,113 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:59,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:59,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,113 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:59,113 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:59,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:54:59,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:59,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:59,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:59,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/57ea68d0ece1412c8f8c5188bf0e1b1b 2023-04-07 01:54:59,213 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:59,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:59,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:59,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:59,217 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:59,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:59,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:59,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,219 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:59,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:59,221 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,221 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:59,221 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:59,221 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:59,221 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:59,226 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,226 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b5e6a9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:59,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:59,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,230 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:59,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:59,233 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:59,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:59,235 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:59,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:59,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:59,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:59,236 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:59,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,240 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,244 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:59,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,245 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:59,245 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:59,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832559259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,345 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:59,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:59,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:59,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:59,349 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:59,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:59,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:59,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:59,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:59,353 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,355 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc5defe[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:59,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:59,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,358 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:59,358 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,358 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:59,359 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:59,362 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:59,362 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:59,362 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:59,362 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:59,362 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:59,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832559363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,366 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:59,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,370 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:59,370 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:59,471 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:59,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:59,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:59,474 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:59,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:59,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:59,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,476 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:59,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:59,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:59,479 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:59,479 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:59,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:59,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd7e40d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:59,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:59,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,484 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:59,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:59,485 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,485 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:59,488 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:59,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:59,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:59,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:59,488 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:59,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,491 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:59,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,495 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:59,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,495 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,495 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832559564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,595 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:59,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:59,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:59,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:59,599 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:59,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:59,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:59,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:59,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:59,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:54:59,604 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7992e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:54:59,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:54:59,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:54:59,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:54:59,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:54:59,614 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:54:59,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:54:59,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:54:59,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:54:59,615 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:54:59,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:54:59,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:54:59,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:54:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:54:59,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,621 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:54:59,621 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:54:59,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/fcda182b4d3c413586690b54a2f1a5d2 2023-04-07 01:54:59,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/a7e7e42f8c6d48d8879a8f54098391e2 2023-04-07 01:54:59,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/57ea68d0ece1412c8f8c5188bf0e1b1b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b 2023-04-07 01:54:59,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b, entries=150, sequenceid=520, filesize=11.9 K 2023-04-07 01:54:59,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/fcda182b4d3c413586690b54a2f1a5d2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2 2023-04-07 01:54:59,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2, entries=150, sequenceid=520, filesize=11.9 K 2023-04-07 01:54:59,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/a7e7e42f8c6d48d8879a8f54098391e2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2 2023-04-07 01:54:59,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2, entries=150, sequenceid=520, filesize=11.9 K 2023-04-07 01:54:59,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ff332f26550df115caee283ff76382bb in 471ms, sequenceid=520, compaction requested=true 2023-04-07 01:54:59,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:59,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:54:59,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:59,655 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:59,655 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:54:59,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:59,656 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:59,656 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:54:59,656 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:54:59,656 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,656 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:54:59,656 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7d97a6d7e08437c8eec2e0dc4011fca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=36.5 K 2023-04-07 01:54:59,656 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,656 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/dae637f7d95b4fc5b43ad017d08c9d2c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=36.5 K 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c7d97a6d7e08437c8eec2e0dc4011fca, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680832497648 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dae637f7d95b4fc5b43ad017d08c9d2c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680832497648 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0ab31dbc87ce4b6ebcb5d20d7d901c7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832497764 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 30fcbd7c83cd4ed0bc411ec80f41eaca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832497764 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 57ea68d0ece1412c8f8c5188bf0e1b1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:54:59,657 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fcda182b4d3c413586690b54a2f1a5d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:54:59,664 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:59,664 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:59,673 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/554ab79be8404dea846d2124c301a762 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/554ab79be8404dea846d2124c301a762 2023-04-07 01:54:59,676 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 554ab79be8404dea846d2124c301a762(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:54:59,676 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:54:59,676 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=13, startTime=1680832499655; duration=0sec 2023-04-07 01:54:59,677 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:54:59,677 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:54:59,678 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:54:59,678 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:54:59,678 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,678 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b940a79086da46db8d9cb212c860bc3f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.4 K 2023-04-07 01:54:59,678 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b940a79086da46db8d9cb212c860bc3f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680832496979 2023-04-07 01:54:59,679 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 52dffc8858014b12a6521a0e4b743cae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680832497648 2023-04-07 01:54:59,679 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 79cefa46c85a48549483ba028d9a774f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680832497764 2023-04-07 01:54:59,679 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a7e7e42f8c6d48d8879a8f54098391e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:54:59,686 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:54:59,721 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:54:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:54:59,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:54:59,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:54:59,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:54:59,724 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:54:59,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:54:59,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:54:59,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:54:59,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:54:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:54:59,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:54:59,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:54:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:54:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:54:59,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:54:59,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:54:59,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:54:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:54:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:54:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:54:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:54:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:54:59,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:54:59,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:54:59,729 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:54:59,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:54:59,730 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,730 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:54:59,730 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,730 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:54:59,730 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:54:59,742 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/43c5e4a6fa3e40c39f02f30ac31ced8e 2023-04-07 01:54:59,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:54:59,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:54:59,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832559891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832559899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832559899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:54:59,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:54:59,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832559994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,075 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/bdc51b021e0c48f2a02b6f4eb725c34b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bdc51b021e0c48f2a02b6f4eb725c34b 2023-04-07 01:55:00,079 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into bdc51b021e0c48f2a02b6f4eb725c34b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:00,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:00,079 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=13, startTime=1680832499655; duration=0sec 2023-04-07 01:55:00,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:00,100 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/6785efa7863c4a0c84ad0a6d9599d9a0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6785efa7863c4a0c84ad0a6d9599d9a0 2023-04-07 01:55:00,104 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into 6785efa7863c4a0c84ad0a6d9599d9a0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:00,104 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:00,104 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=12, startTime=1680832499655; duration=0sec 2023-04-07 01:55:00,104 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:00,153 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bf0bd9327f4049fdaa3d238ad50a001a 2023-04-07 01:55:00,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:00,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832560198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832560501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,563 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/6a5bc992068947728fa1f91b58000ff3 2023-04-07 01:55:00,568 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/43c5e4a6fa3e40c39f02f30ac31ced8e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e 2023-04-07 01:55:00,571 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e, entries=150, sequenceid=541, filesize=11.9 K 2023-04-07 01:55:00,572 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/bf0bd9327f4049fdaa3d238ad50a001a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a 2023-04-07 01:55:00,576 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a, entries=150, sequenceid=541, filesize=11.9 K 2023-04-07 01:55:00,576 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/6a5bc992068947728fa1f91b58000ff3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3 2023-04-07 01:55:00,600 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3, entries=150, sequenceid=541, filesize=11.9 K 2023-04-07 01:55:00,601 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ff332f26550df115caee283ff76382bb in 872ms, sequenceid=541, compaction requested=false 2023-04-07 01:55:00,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:00,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:00,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:00,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:00,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1cb13d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:00,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:00,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,610 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:00,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:00,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:00,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:00,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:00,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:00,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:00,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:00,615 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:00,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:00,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:00,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:00,615 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:00,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:00,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:00,618 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:00,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:00,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:00,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:00,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:00,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:00,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:00,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:00,622 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:00,622 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:00,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:00,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,722 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:00,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:00,725 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:00,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:00,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:00,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:00,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:00,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:00,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:00,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:00,727 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:00,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:00,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:00,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:00,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:00,729 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:00,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:01,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832561050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,136 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/d608d51e722246c5b55ce652fca51506 2023-04-07 01:55:01,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832561153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832561356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,551 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0818b903090d4500bddc8daba5f6b4b8 2023-04-07 01:55:01,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832561661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832561911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,913 DEBUG [Thread-1670] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:01,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832561920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:01,923 DEBUG [Thread-1674] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:02,033 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/4f2aa99e6e3c468cb7a8aacfa4405523 2023-04-07 01:55:02,036 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/d608d51e722246c5b55ce652fca51506 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506 2023-04-07 01:55:02,040 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506, entries=150, sequenceid=555, filesize=11.9 K 2023-04-07 01:55:02,040 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0818b903090d4500bddc8daba5f6b4b8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8 2023-04-07 01:55:02,044 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8, entries=150, sequenceid=555, filesize=11.9 K 2023-04-07 01:55:02,045 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/4f2aa99e6e3c468cb7a8aacfa4405523 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523 2023-04-07 01:55:02,048 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523, entries=150, sequenceid=555, filesize=11.9 K 2023-04-07 01:55:02,048 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ff332f26550df115caee283ff76382bb in 1319ms, sequenceid=555, compaction requested=true 2023-04-07 01:55:02,049 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:02,049 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2fac8e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,058 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,058 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,066 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,066 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,166 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:02,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832501038, max timestamp=1680832502166Num uniques 859; 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832501038, max timestamp=1680832502166Num uniques 859; 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832501038, max timestamp=1680832502166Num uniques 859; 2023-04-07 01:55:02,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:02,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,171 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,174 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,174 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,174 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,174 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,176 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,176 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc 2023-04-07 01:55:02,176 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,176 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ad64a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,183 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,185 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,188 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,188 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,198 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,198 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,198 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832502211, max timestamp=1680832502221Num uniques 859; 2023-04-07 01:55:02,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832502211, max timestamp=1680832502221Num uniques 859; 2023-04-07 01:55:02,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832502211, max timestamp=1680832502221Num uniques 859; 2023-04-07 01:55:02,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:02,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832502263, max timestamp=1680832502274Num uniques 1018; 2023-04-07 01:55:02,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:02,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832502263, max timestamp=1680832502274Num uniques 1018; 2023-04-07 01:55:02,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832502263, max timestamp=1680832502274Num uniques 1018; 2023-04-07 01:55:02,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:02,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:02,298 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,302 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,304 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,305 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,305 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,305 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,305 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34084680[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,311 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,322 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,322 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832502326, max timestamp=1680832502336Num uniques 1027; 2023-04-07 01:55:02,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832502326, max timestamp=1680832502336Num uniques 1027; 2023-04-07 01:55:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832502326, max timestamp=1680832502336Num uniques 1027; 2023-04-07 01:55:02,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:02,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832502381, max timestamp=1680832502402Num uniques 1036; 2023-04-07 01:55:02,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832502381, max timestamp=1680832502402Num uniques 1036; 2023-04-07 01:55:02,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:02,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:02,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832502381, max timestamp=1680832502402Num uniques 1036; 2023-04-07 01:55:02,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:02,422 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,426 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,428 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,429 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a739c65[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,436 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,438 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,439 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,439 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,439 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,440 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,444 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,444 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,444 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832502448, max timestamp=1680832502467Num uniques 1045; 2023-04-07 01:55:02,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:02,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832502448, max timestamp=1680832502467Num uniques 1045; 2023-04-07 01:55:02,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:02,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832502448, max timestamp=1680832502467Num uniques 1045; 2023-04-07 01:55:02,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:02,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832502501, max timestamp=1680832502528Num uniques 1054; 2023-04-07 01:55:02,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:02,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832502501, max timestamp=1680832502528Num uniques 1054; 2023-04-07 01:55:02,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832502501, max timestamp=1680832502528Num uniques 1054; 2023-04-07 01:55:02,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:02,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:02,545 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,548 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,548 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,548 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,548 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4f0079[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,557 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,558 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,562 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,563 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,573 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,573 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832502545, max timestamp=1680832502575Num uniques 1013; 2023-04-07 01:55:02,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:02,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832502545, max timestamp=1680832502575Num uniques 1013; 2023-04-07 01:55:02,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832502545, max timestamp=1680832502575Num uniques 1013; 2023-04-07 01:55:02,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:02,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:02,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/255862264f95479fa10ae50d5287e19a 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832502618, max timestamp=1680832502636Num uniques 1022; 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832502618, max timestamp=1680832502636Num uniques 1022; 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832502618, max timestamp=1680832502636Num uniques 1022; 2023-04-07 01:55:02,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:55:02,673 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,677 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,683 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,683 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,683 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,683 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43795b1c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,687 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,688 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832502675, max timestamp=1680832502689Num uniques 1031; 2023-04-07 01:55:02,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,691 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,691 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832502675, max timestamp=1680832502689Num uniques 1031; 2023-04-07 01:55:02,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832502675, max timestamp=1680832502689Num uniques 1031; 2023-04-07 01:55:02,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:55:02,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:55:02,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,699 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,700 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,700 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832502715, max timestamp=1680832502741Num uniques 1040; 2023-04-07 01:55:02,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:55:02,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832502715, max timestamp=1680832502741Num uniques 1040; 2023-04-07 01:55:02,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:55:02,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832502715, max timestamp=1680832502741Num uniques 1040; 2023-04-07 01:55:02,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:55:02,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832502784, max timestamp=1680832502796Num uniques 1049; 2023-04-07 01:55:02,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:55:02,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832502784, max timestamp=1680832502796Num uniques 1049; 2023-04-07 01:55:02,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832502784, max timestamp=1680832502796Num uniques 1049; 2023-04-07 01:55:02,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:55:02,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:55:02,800 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,803 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,803 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,803 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,803 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,808 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,808 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,808 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,808 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b339b5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,818 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,818 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,821 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:02,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,826 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,826 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832502843, max timestamp=1680832502852Num uniques 1058; 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832502843, max timestamp=1680832502852Num uniques 1058; 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832502843, max timestamp=1680832502852Num uniques 1058; 2023-04-07 01:55:02,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832502893, max timestamp=1680832502906Num uniques 1017; 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832502893, max timestamp=1680832502906Num uniques 1017; 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832502893, max timestamp=1680832502906Num uniques 1017; 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-07 01:55:02,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-07 01:55:02,926 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:02,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:02,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:02,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:02,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:02,930 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:02,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:02,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:02,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:02,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:02,938 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,938 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:02,938 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:02,938 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:02,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e51bd2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:02,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:02,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:02,945 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,945 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,946 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:02,948 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:02,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:02,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:02,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:02,948 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:02,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:02,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:02,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,951 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:02,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:02,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:02,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:02,954 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:02,954 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:02,954 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:02,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832502958, max timestamp=1680832502973Num uniques 1076; 2023-04-07 01:55:02,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:55:02,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832502958, max timestamp=1680832502973Num uniques 1076; 2023-04-07 01:55:02,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:55:02,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832502958, max timestamp=1680832502973Num uniques 1076; 2023-04-07 01:55:02,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 01:55:02,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/02bb8715c060418e8d665a27fa82fef3 2023-04-07 01:55:03,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc 2023-04-07 01:55:03,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc, entries=150, sequenceid=576, filesize=11.9 K 2023-04-07 01:55:03,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/255862264f95479fa10ae50d5287e19a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a 2023-04-07 01:55:03,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a, entries=150, sequenceid=576, filesize=11.9 K 2023-04-07 01:55:03,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/02bb8715c060418e8d665a27fa82fef3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3 2023-04-07 01:55:03,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3, entries=150, sequenceid=576, filesize=11.9 K 2023-04-07 01:55:03,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=100.63 KB/103050 for ff332f26550df115caee283ff76382bb in 852ms, sequenceid=576, compaction requested=true 2023-04-07 01:55:03,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:03,020 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:03,020 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:03,021 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:03,021 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:03,021 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:55:03,021 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:55:03,022 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,022 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,022 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bdc51b021e0c48f2a02b6f4eb725c34b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.5 K 2023-04-07 01:55:03,022 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/554ab79be8404dea846d2124c301a762, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.5 K 2023-04-07 01:55:03,022 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting bdc51b021e0c48f2a02b6f4eb725c34b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:55:03,022 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 554ab79be8404dea846d2124c301a762, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:55:03,022 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 43c5e4a6fa3e40c39f02f30ac31ced8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680832499250 2023-04-07 01:55:03,023 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bf0bd9327f4049fdaa3d238ad50a001a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680832499250 2023-04-07 01:55:03,023 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d608d51e722246c5b55ce652fca51506, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680832499884 2023-04-07 01:55:03,023 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0818b903090d4500bddc8daba5f6b4b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680832499884 2023-04-07 01:55:03,023 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 81fe1ff34b6e4bd69b8fbdfe56a3abdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:03,024 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 255862264f95479fa10ae50d5287e19a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:03,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:03,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=107.34 KB heapSize=303.70 KB 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:03,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:03,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/4b6c91d971ae40fa9e2b9dbe5490277a 2023-04-07 01:55:03,049 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#303 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:03,050 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#304 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:03,054 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,064 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:03,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 545 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832563073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,078 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/3cf12ac148cf434b8546912f407967b3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3cf12ac148cf434b8546912f407967b3 2023-04-07 01:55:03,084 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into 3cf12ac148cf434b8546912f407967b3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:03,084 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:03,084 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=12, startTime=1680832503020; duration=0sec 2023-04-07 01:55:03,084 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:03,084 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:03,086 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:03,086 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:55:03,086 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,086 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6785efa7863c4a0c84ad0a6d9599d9a0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.5 K 2023-04-07 01:55:03,086 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6785efa7863c4a0c84ad0a6d9599d9a0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680832498871 2023-04-07 01:55:03,087 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6a5bc992068947728fa1f91b58000ff3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680832499250 2023-04-07 01:55:03,087 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4f2aa99e6e3c468cb7a8aacfa4405523, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680832499884 2023-04-07 01:55:03,087 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 02bb8715c060418e8d665a27fa82fef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:03,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,090 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fc811f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,094 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,096 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:03,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,101 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,102 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,105 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,105 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,106 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,106 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,107 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,110 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,110 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,111 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,111 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 547 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832563177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,211 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,215 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,230 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f548f67[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,232 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,232 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,238 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,238 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,238 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,239 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,239 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,241 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,241 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,241 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,243 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,245 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,248 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,248 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,248 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,348 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,351 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,353 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,353 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,353 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,353 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38675a9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,359 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,361 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,361 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,361 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,362 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,366 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,366 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,366 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,366 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,366 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,369 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,373 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,373 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,373 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:03,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 549 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832563380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,473 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,477 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,477 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,479 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/1008f213d1ca4a81b875e731dc1db0b4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1008f213d1ca4a81b875e731dc1db0b4 2023-04-07 01:55:03,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,480 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,480 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,482 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,486 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 1008f213d1ca4a81b875e731dc1db0b4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:03,486 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:03,486 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=12, startTime=1680832503020; duration=0sec 2023-04-07 01:55:03,486 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:03,486 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,487 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5866c580[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,488 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,489 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,492 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,496 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/1b2fa4d24aad45dfb98ab0d5a0fb6687 2023-04-07 01:55:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,498 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,500 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,501 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,501 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,502 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,523 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/6bf4602822f34dceab30557b25cb3337 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6bf4602822f34dceab30557b25cb3337 2023-04-07 01:55:03,529 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into 6bf4602822f34dceab30557b25cb3337(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:03,529 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:03,529 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=12, startTime=1680832503020; duration=0sec 2023-04-07 01:55:03,529 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:03,602 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,605 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,605 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,607 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,608 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,609 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,609 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,609 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,609 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,613 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad053ea[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,617 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,620 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,620 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,622 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,624 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,627 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,627 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:03,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 551 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832563685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,727 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,731 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,731 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,732 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,732 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,733 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,734 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,734 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,734 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,734 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,736 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,736 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,737 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f4d860[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,739 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,742 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,742 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,744 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,744 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,744 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,750 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,750 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,850 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,856 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,858 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,858 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,859 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,859 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:03,863 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21efdcca[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:03,865 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:03,865 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,866 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:03,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:03,871 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:03,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:03,874 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:03,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:03,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:03,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:03,874 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:03,881 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,881 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,881 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:03,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:03,883 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:03,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:03,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:03,885 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:03,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:03,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:03,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:03,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,886 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:03,886 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:03,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/b0c9d2307080474c9e5a1f70bde503a0 2023-04-07 01:55:03,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/4b6c91d971ae40fa9e2b9dbe5490277a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a 2023-04-07 01:55:03,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a, entries=200, sequenceid=832, filesize=14.3 K 2023-04-07 01:55:03,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/1b2fa4d24aad45dfb98ab0d5a0fb6687 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687 2023-04-07 01:55:03,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687, entries=150, sequenceid=832, filesize=11.9 K 2023-04-07 01:55:03,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/b0c9d2307080474c9e5a1f70bde503a0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0 2023-04-07 01:55:03,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0, entries=150, sequenceid=832, filesize=11.9 K 2023-04-07 01:55:03,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~303.66 KB/310944, currentSize=73.80 KB/75570 for ff332f26550df115caee283ff76382bb in 923ms, sequenceid=832, compaction requested=false 2023-04-07 01:55:03,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:03,986 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:03,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:03,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:03,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:03,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:03,990 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:03,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:03,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:03,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:03,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:03,992 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:03,992 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:03,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:03,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:03,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:03,993 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:03,994 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:03,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:04,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:04,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:04,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 568 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832564228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:04,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 570 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832564331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,402 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/18724a3e8f794e9dbceb6a3fe11e8279 2023-04-07 01:55:04,413 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a3d3467181f441c8315784ed8afb013 2023-04-07 01:55:04,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:04,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 572 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832564532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,822 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/c69385f2294f4fd7bf3f49d2b9cb4b59 2023-04-07 01:55:04,825 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/18724a3e8f794e9dbceb6a3fe11e8279 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279 2023-04-07 01:55:04,829 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279, entries=150, sequenceid=849, filesize=11.9 K 2023-04-07 01:55:04,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/0a3d3467181f441c8315784ed8afb013 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013 2023-04-07 01:55:04,833 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013, entries=150, sequenceid=849, filesize=11.9 K 2023-04-07 01:55:04,834 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/c69385f2294f4fd7bf3f49d2b9cb4b59 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59 2023-04-07 01:55:04,837 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59, entries=150, sequenceid=849, filesize=11.9 K 2023-04-07 01:55:04,838 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ff332f26550df115caee283ff76382bb in 844ms, sequenceid=849, compaction requested=true 2023-04-07 01:55:04,838 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:04,838 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:04,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:04,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:04,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:04,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:04,839 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:04,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:04,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:04,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:04,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:04,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:04,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:04,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:04,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4b47b2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:04,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:04,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,844 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:04,845 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,845 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:04,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:04,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:04,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/9d3481ea61504507a570fa7f49242b95 2023-04-07 01:55:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:04,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:04,849 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:04,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:04,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:04,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:04,850 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:04,852 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,852 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:04,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,852 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,853 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,854 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:04,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,857 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:04,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:04,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:04,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:04,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:04,857 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,857 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:04,857 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:04,857 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,857 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,857 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:04,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:04,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c6f88136785e490282bb31e20bee475f 2023-04-07 01:55:04,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:04,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 585 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832564873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,957 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:04,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:04,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:04,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:04,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:04,960 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:04,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:04,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:04,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:04,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:04,964 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:04,964 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,964 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:04,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:04,974 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,974 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@571c02b2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:04,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:04,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:04,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 587 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832564976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,978 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:04,978 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:04,978 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:04,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:04,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:04,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:04,983 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:04,983 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:04,983 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:04,983 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:04,983 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:04,986 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,986 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:04,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,987 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:04,988 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:04,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:04,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:04,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:04,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:04,994 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:04,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:04,995 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:04,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:04,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:04,995 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:04,995 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:04,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:05,095 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:05,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:05,098 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:05,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:05,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:05,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:05,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:05,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:05,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:05,102 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,102 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:05,102 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:05,102 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:05,103 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,103 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,103 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601ce7e9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:05,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:05,105 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,112 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:05,112 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,112 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:05,114 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:05,116 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:05,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:05,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:05,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:05,116 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:05,120 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,120 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:05,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,120 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:05,121 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:05,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:05,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:05,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:05,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,125 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:05,125 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,125 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:05,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:05,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 589 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832565180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,225 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:05,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:05,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:05,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:05,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:05,228 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:05,228 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,229 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:05,229 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:05,229 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,229 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,231 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:05,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:05,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:05,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:05,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:05,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:05,233 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:05,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,234 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fca6eac[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:05,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:05,236 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,237 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:05,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:05,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:05,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:05,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:05,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:05,242 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:05,242 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:05,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:05,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:05,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,249 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:05,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:05,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:05,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:05,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:05,249 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:05,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:05,250 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,250 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:05,250 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:05,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/fedc1cb04f6e4694bcffbebfeb753c66 2023-04-07 01:55:05,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/9d3481ea61504507a570fa7f49242b95 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95 2023-04-07 01:55:05,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95, entries=200, sequenceid=868, filesize=14.3 K 2023-04-07 01:55:05,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c6f88136785e490282bb31e20bee475f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f 2023-04-07 01:55:05,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f, entries=150, sequenceid=868, filesize=11.9 K 2023-04-07 01:55:05,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/fedc1cb04f6e4694bcffbebfeb753c66 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66 2023-04-07 01:55:05,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66, entries=150, sequenceid=868, filesize=11.9 K 2023-04-07 01:55:05,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ff332f26550df115caee283ff76382bb in 445ms, sequenceid=868, compaction requested=true 2023-04-07 01:55:05,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:05,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:05,286 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:05,286 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:05,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:05,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:05,287 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54714 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:05,287 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:05,287 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:55:05,287 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:55:05,287 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,287 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,288 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3cf12ac148cf434b8546912f407967b3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=53.4 K 2023-04-07 01:55:05,288 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1008f213d1ca4a81b875e731dc1db0b4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.7 K 2023-04-07 01:55:05,288 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3cf12ac148cf434b8546912f407967b3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:05,288 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1008f213d1ca4a81b875e731dc1db0b4, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:05,288 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4b6c91d971ae40fa9e2b9dbe5490277a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680832502999 2023-04-07 01:55:05,288 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1b2fa4d24aad45dfb98ab0d5a0fb6687, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680832502999 2023-04-07 01:55:05,289 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0a3d3467181f441c8315784ed8afb013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1680832503049 2023-04-07 01:55:05,289 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 18724a3e8f794e9dbceb6a3fe11e8279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1680832503049 2023-04-07 01:55:05,289 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c6f88136785e490282bb31e20bee475f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:05,290 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9d3481ea61504507a570fa7f49242b95, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:05,298 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:05,300 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:05,350 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:05,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:05,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:05,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:05,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:05,354 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:05,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:05,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:05,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,354 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,356 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:05,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:05,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:05,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:05,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:05,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:05,358 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:55:05,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:05,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:05,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:05,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:05,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:05,359 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:05,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:05,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:05,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 607 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832565544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 609 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832565649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,710 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/3fc2b562a4194644b47dfc41b3f2e01a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/3fc2b562a4194644b47dfc41b3f2e01a 2023-04-07 01:55:05,714 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/6593c79589f1477680fc9149e7e5d0a4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/6593c79589f1477680fc9149e7e5d0a4 2023-04-07 01:55:05,716 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 3fc2b562a4194644b47dfc41b3f2e01a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:05,716 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:05,716 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=12, startTime=1680832505286; duration=0sec 2023-04-07 01:55:05,716 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:05,716 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:05,718 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:05,718 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:55:05,718 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:05,719 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6bf4602822f34dceab30557b25cb3337, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=48.6 K 2023-04-07 01:55:05,719 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6bf4602822f34dceab30557b25cb3337, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1680832501038 2023-04-07 01:55:05,719 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into 6593c79589f1477680fc9149e7e5d0a4(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:05,720 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:05,720 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b0c9d2307080474c9e5a1f70bde503a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680832502999 2023-04-07 01:55:05,720 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=12, startTime=1680832505286; duration=0sec 2023-04-07 01:55:05,720 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:05,720 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c69385f2294f4fd7bf3f49d2b9cb4b59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1680832503049 2023-04-07 01:55:05,720 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fedc1cb04f6e4694bcffbebfeb753c66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:05,727 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:05,766 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ae81799b07ab4c398729ff1eb8174b49 2023-04-07 01:55:05,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 611 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832565852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60746 deadline: 1680832565921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,924 DEBUG [Thread-1670] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:05,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:05,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1680832565962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:05,967 DEBUG [Thread-1674] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:06,135 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/b46d0701dcf44126beb5c3a2fb708adb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b46d0701dcf44126beb5c3a2fb708adb 2023-04-07 01:55:06,139 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into b46d0701dcf44126beb5c3a2fb708adb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:06,139 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:06,139 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=12, startTime=1680832505286; duration=0sec 2023-04-07 01:55:06,139 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:06,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:06,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 613 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832566156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,181 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c6e09807c2e2463ebce31b52f2fd014d 2023-04-07 01:55:06,596 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/40b47830b85e46e89273914ae5c89ea1 2023-04-07 01:55:06,599 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/ae81799b07ab4c398729ff1eb8174b49 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49 2023-04-07 01:55:06,607 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49, entries=150, sequenceid=881, filesize=11.9 K 2023-04-07 01:55:06,608 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c6e09807c2e2463ebce31b52f2fd014d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d 2023-04-07 01:55:06,611 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d, entries=150, sequenceid=881, filesize=11.9 K 2023-04-07 01:55:06,612 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/40b47830b85e46e89273914ae5c89ea1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1 2023-04-07 01:55:06,615 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1, entries=150, sequenceid=881, filesize=11.9 K 2023-04-07 01:55:06,615 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ff332f26550df115caee283ff76382bb in 1257ms, sequenceid=881, compaction requested=false 2023-04-07 01:55:06,615 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:06,615 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:06,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:06,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:06,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:06,616 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:06,618 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:06,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa9d830[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:06,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:06,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:06,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:06,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:06,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:06,623 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:06,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:06,625 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:06,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:06,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:06,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:06,626 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:06,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,627 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,628 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:06,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:06,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,633 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,633 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,633 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,633 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:06,633 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:06,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:06,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:06,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:55:06,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:06,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:06,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:06,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:06,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/c2b37265b8214a9c8f21ac72e98d24bc 2023-04-07 01:55:06,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:06,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 625 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832566688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,733 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:06,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:06,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:06,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:06,736 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:06,736 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:06,736 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:06,736 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:06,737 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,737 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,738 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:06,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:06,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,739 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:06,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:06,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:06,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:06,740 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:06,740 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:06,740 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:06,741 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:06,741 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:06,741 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:06,741 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:06,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:06,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,743 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:06,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d175645[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:06,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:06,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,748 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:06,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:06,751 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:06,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:06,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:06,752 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:06,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:06,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:06,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:06,752 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:06,755 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,755 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,755 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,756 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:06,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:06,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:06,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:06,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,765 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,765 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:06,765 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:06,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:06,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 627 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832566790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,865 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:06,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:06,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:06,872 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:06,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:06,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:06,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:06,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,874 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:06,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:06,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,875 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,875 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:06,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:06,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:06,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:06,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:06,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:06,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:06,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,888 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:06,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab3d9a1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:06,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:06,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,891 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:06,891 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:06,891 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:06,894 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:06,897 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:06,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:06,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:06,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:06,897 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:06,899 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,899 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,899 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:06,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:06,901 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:06,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:06,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:06,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:06,904 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:06,904 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:06,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:06,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:06,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:06,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:06,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:06,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 629 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832566993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,004 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:07,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:07,008 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:07,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:07,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:07,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:07,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,010 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:07,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:07,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:07,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:07,012 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,012 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:07,012 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:07,012 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:07,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,015 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:07,015 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,015 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e07b55[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:07,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:07,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,018 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,018 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:07,018 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:07,018 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:07,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:07,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:07,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:07,022 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:07,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:07,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:07,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:07,022 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:07,024 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,024 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:07,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,024 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:07,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:07,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,025 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:07,028 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:07,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:07,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:07,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:07,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:07,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,029 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:07,029 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:07,029 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/a77100f7468048bbbbe70521319b7b6b 2023-04-07 01:55:07,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/71ac76bafe6a42189950a4ab8dafb659 2023-04-07 01:55:07,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/c2b37265b8214a9c8f21ac72e98d24bc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc 2023-04-07 01:55:07,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc, entries=200, sequenceid=904, filesize=14.3 K 2023-04-07 01:55:07,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/a77100f7468048bbbbe70521319b7b6b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b 2023-04-07 01:55:07,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b, entries=150, sequenceid=904, filesize=11.9 K 2023-04-07 01:55:07,114 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 01:55:07,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/71ac76bafe6a42189950a4ab8dafb659 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659 2023-04-07 01:55:07,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659, entries=150, sequenceid=904, filesize=11.9 K 2023-04-07 01:55:07,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ff332f26550df115caee283ff76382bb in 454ms, sequenceid=904, compaction requested=true 2023-04-07 01:55:07,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:07,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:07,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:07,119 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:07,119 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:07,119 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:07,119 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:07,120 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:55:07,120 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,120 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/6593c79589f1477680fc9149e7e5d0a4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=39.3 K 2023-04-07 01:55:07,120 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6593c79589f1477680fc9149e7e5d0a4, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:07,120 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:07,121 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ae81799b07ab4c398729ff1eb8174b49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1680832504854 2023-04-07 01:55:07,121 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:55:07,121 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,121 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/3fc2b562a4194644b47dfc41b3f2e01a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=36.9 K 2023-04-07 01:55:07,121 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c2b37265b8214a9c8f21ac72e98d24bc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505527 2023-04-07 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3fc2b562a4194644b47dfc41b3f2e01a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c6e09807c2e2463ebce31b52f2fd014d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1680832504854 2023-04-07 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a77100f7468048bbbbe70521319b7b6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505538 2023-04-07 01:55:07,129 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:07,129 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:07,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:07,130 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#324 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:07,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:07,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:07,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:07,135 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:07,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:07,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:07,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:07,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,137 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:07,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:07,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:07,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:07,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:07,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:07,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:07,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:07,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:07,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:07,140 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,140 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. started... 2023-04-07 01:55:07,140 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:07,141 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:07,143 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/931a03d7bed34bb1909abfc39e0421d5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/931a03d7bed34bb1909abfc39e0421d5 2023-04-07 01:55:07,143 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/bd13c657d26a4b278bea3371ba3a944a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bd13c657d26a4b278bea3371ba3a944a 2023-04-07 01:55:07,152 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into bd13c657d26a4b278bea3371ba3a944a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:07,152 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:07,152 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=13, startTime=1680832507119; duration=0sec 2023-04-07 01:55:07,152 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:07,153 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:07,154 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:07,154 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:55:07,154 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:07,154 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b46d0701dcf44126beb5c3a2fb708adb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=36.9 K 2023-04-07 01:55:07,154 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b46d0701dcf44126beb5c3a2fb708adb, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1680832504212 2023-04-07 01:55:07,154 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 40b47830b85e46e89273914ae5c89ea1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1680832504854 2023-04-07 01:55:07,155 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 71ac76bafe6a42189950a4ab8dafb659, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505538 2023-04-07 01:55:07,157 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 931a03d7bed34bb1909abfc39e0421d5(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:07,157 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:07,157 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=13, startTime=1680832507119; duration=0sec 2023-04-07 01:55:07,157 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:07,161 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#326 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:07,177 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/d24c9047ee384abca4cf2e55259a2484 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d24c9047ee384abca4cf2e55259a2484 2023-04-07 01:55:07,183 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into d24c9047ee384abca4cf2e55259a2484(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:07,183 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:07,183 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=13, startTime=1680832507119; duration=0sec 2023-04-07 01:55:07,183 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:07,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:07,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. as already flushing 2023-04-07 01:55:07,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:07,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 648 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832567353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,364 DEBUG [Thread-1681] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:61971 2023-04-07 01:55:07,364 DEBUG [Thread-1681] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:07,369 DEBUG [Thread-1677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:61971 2023-04-07 01:55:07,369 DEBUG [Thread-1677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:07,371 DEBUG [Thread-1685] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:61971 2023-04-07 01:55:07,371 DEBUG [Thread-1679] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:61971 2023-04-07 01:55:07,371 DEBUG [Thread-1683] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:61971 2023-04-07 01:55:07,371 DEBUG [Thread-1679] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:07,371 DEBUG [Thread-1685] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:07,371 DEBUG [Thread-1683] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:07,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:07,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 650 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832567456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,565 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/d436957ff4f748c28c49ec0ab026002d 2023-04-07 01:55:07,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 652 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832567659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:07,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 654 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60708 deadline: 1680832567962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:07,974 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/4fd27c6599ad4f4799f7756bd1d44aac 2023-04-07 01:55:08,383 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/0f8deab68d184cbabb546d6e9111346b 2023-04-07 01:55:08,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/d436957ff4f748c28c49ec0ab026002d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d 2023-04-07 01:55:08,389 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d, entries=150, sequenceid=916, filesize=11.9 K 2023-04-07 01:55:08,390 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/4fd27c6599ad4f4799f7756bd1d44aac as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac 2023-04-07 01:55:08,392 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac, entries=150, sequenceid=916, filesize=11.9 K 2023-04-07 01:55:08,393 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/0f8deab68d184cbabb546d6e9111346b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b 2023-04-07 01:55:08,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b, entries=150, sequenceid=916, filesize=11.9 K 2023-04-07 01:55:08,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ff332f26550df115caee283ff76382bb in 1256ms, sequenceid=916, compaction requested=false 2023-04-07 01:55:08,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:08,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:08,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:08,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:08,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:08,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:08,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:08,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:08,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:08,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@126c45e1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:08,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:08,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,401 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:08,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:08,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:08,404 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:08,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:08,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:08,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:08,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:08,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:08,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:08,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:08,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:08,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:08,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:08,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:08,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:08,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:08,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:08,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:08,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:08,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:08,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:08,414 DEBUG [Thread-1676] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:08,414 DEBUG [Thread-1676] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:08,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on ff332f26550df115caee283ff76382bb 2023-04-07 01:55:08,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 01:55:08,467 DEBUG [Thread-1668] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:61971 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:08,467 DEBUG [Thread-1668] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:08,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:08,514 DEBUG [Thread-1676] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:08,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:08,528 DEBUG [Thread-1672] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:61971 2023-04-07 01:55:08,528 DEBUG [Thread-1672] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:08,528 DEBUG [Thread-1666] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:61971 2023-04-07 01:55:08,529 DEBUG [Thread-1666] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:08,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/7226a2a3481e42c49fa197d6422cc463 2023-04-07 01:55:09,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/5a88f06a52ff4b5ebd7144e066c4ff04 2023-04-07 01:55:09,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=940 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/19b1f903023b472ab1b28664af5b867d 2023-04-07 01:55:09,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/7226a2a3481e42c49fa197d6422cc463 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463 2023-04-07 01:55:09,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463, entries=150, sequenceid=940, filesize=11.9 K 2023-04-07 01:55:09,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/5a88f06a52ff4b5ebd7144e066c4ff04 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04 2023-04-07 01:55:09,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04, entries=150, sequenceid=940, filesize=11.9 K 2023-04-07 01:55:09,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/19b1f903023b472ab1b28664af5b867d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d 2023-04-07 01:55:09,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d, entries=150, sequenceid=940, filesize=11.9 K 2023-04-07 01:55:09,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=13.42 KB/13740 for ff332f26550df115caee283ff76382bb in 1239ms, sequenceid=940, compaction requested=true 2023-04-07 01:55:09,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:09,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:09,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:09,705 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:09,705 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:09,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/A is initiating minor compaction (all files) 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/B is initiating minor compaction (all files) 2023-04-07 01:55:09,706 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/A in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:09,706 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/B in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:09,706 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bd13c657d26a4b278bea3371ba3a944a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=37.0 K 2023-04-07 01:55:09,706 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/931a03d7bed34bb1909abfc39e0421d5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=37.0 K 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting bd13c657d26a4b278bea3371ba3a944a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505538 2023-04-07 01:55:09,706 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 931a03d7bed34bb1909abfc39e0421d5, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505538 2023-04-07 01:55:09,707 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d436957ff4f748c28c49ec0ab026002d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1680832506666 2023-04-07 01:55:09,707 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4fd27c6599ad4f4799f7756bd1d44aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1680832506666 2023-04-07 01:55:09,707 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7226a2a3481e42c49fa197d6422cc463, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680832507341 2023-04-07 01:55:09,707 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5a88f06a52ff4b5ebd7144e066c4ff04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680832507341 2023-04-07 01:55:09,717 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#A#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:09,717 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#B#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:10,129 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/15d5298ce93246f48203b2333158a518 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15d5298ce93246f48203b2333158a518 2023-04-07 01:55:10,129 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/c7331c2d6dc547f7b451ed27ea95bcd0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7331c2d6dc547f7b451ed27ea95bcd0 2023-04-07 01:55:10,133 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/B of ff332f26550df115caee283ff76382bb into 15d5298ce93246f48203b2333158a518(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:10,133 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:10,133 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/B, priority=13, startTime=1680832509705; duration=0sec 2023-04-07 01:55:10,133 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:10,133 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:10,133 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/A of ff332f26550df115caee283ff76382bb into c7331c2d6dc547f7b451ed27ea95bcd0(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:10,133 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:10,133 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/A, priority=13, startTime=1680832509705; duration=0sec 2023-04-07 01:55:10,134 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:10,134 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:10,134 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): ff332f26550df115caee283ff76382bb/C is initiating minor compaction (all files) 2023-04-07 01:55:10,134 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ff332f26550df115caee283ff76382bb/C in TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:10,134 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d24c9047ee384abca4cf2e55259a2484, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp, totalSize=37.0 K 2023-04-07 01:55:10,135 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d24c9047ee384abca4cf2e55259a2484, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1680832505538 2023-04-07 01:55:10,135 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0f8deab68d184cbabb546d6e9111346b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1680832506666 2023-04-07 01:55:10,135 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 19b1f903023b472ab1b28664af5b867d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=940, earliestPutTs=1680832507341 2023-04-07 01:55:10,140 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): ff332f26550df115caee283ff76382bb#C#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:10,548 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/0678c63c48944aa5b7a912e17727be49 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0678c63c48944aa5b7a912e17727be49 2023-04-07 01:55:10,552 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ff332f26550df115caee283ff76382bb/C of ff332f26550df115caee283ff76382bb into 0678c63c48944aa5b7a912e17727be49(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:10,552 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:10,552 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb., storeName=ff332f26550df115caee283ff76382bb/C, priority=13, startTime=1680832509705; duration=0sec 2023-04-07 01:55:10,552 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:15,985 DEBUG [Thread-1670] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:61971 2023-04-07 01:55:15,985 DEBUG [Thread-1670] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:16,057 DEBUG [Thread-1674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:61971 2023-04-07 01:55:16,057 DEBUG [Thread-1674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 521 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4898 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4893 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4919 2023-04-07 01:55:16,057 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4870 2023-04-07 01:55:16,058 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4887 2023-04-07 01:55:16,058 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:55:16,058 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:55:16,058 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:61971 2023-04-07 01:55:16,058 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:16,058 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:55:16,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:55:16,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:16,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:16,062 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832516062"}]},"ts":"1680832516062"} 2023-04-07 01:55:16,063 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:55:16,065 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:55:16,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, UNASSIGN}] 2023-04-07 01:55:16,067 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, UNASSIGN 2023-04-07 01:55:16,067 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ff332f26550df115caee283ff76382bb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:16,067 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832516067"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832516067"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832516067"}]},"ts":"1680832516067"} 2023-04-07 01:55:16,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:16,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:16,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ff332f26550df115caee283ff76382bb 2023-04-07 01:55:16,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ff332f26550df115caee283ff76382bb, disabling compactions & flushes 2023-04-07 01:55:16,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. after waiting 0 ms 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:16,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ff332f26550df115caee283ff76382bb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=A 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=B 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ff332f26550df115caee283ff76382bb, store=C 2023-04-07 01:55:16,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:16,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/e4120e64059f4d3697d965f4d7ea824c 2023-04-07 01:55:16,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:16,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c8258148468c490284093184f93b9405 2023-04-07 01:55:16,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:17,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/87ee25557eb648ff8b069a5c0c58b0ba 2023-04-07 01:55:17,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/A/e4120e64059f4d3697d965f4d7ea824c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e4120e64059f4d3697d965f4d7ea824c 2023-04-07 01:55:17,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e4120e64059f4d3697d965f4d7ea824c, entries=150, sequenceid=950, filesize=11.9 K 2023-04-07 01:55:17,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/B/c8258148468c490284093184f93b9405 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c8258148468c490284093184f93b9405 2023-04-07 01:55:17,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c8258148468c490284093184f93b9405, entries=150, sequenceid=950, filesize=11.9 K 2023-04-07 01:55:17,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/.tmp/C/87ee25557eb648ff8b069a5c0c58b0ba as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/87ee25557eb648ff8b069a5c0c58b0ba 2023-04-07 01:55:17,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/87ee25557eb648ff8b069a5c0c58b0ba, entries=150, sequenceid=950, filesize=11.9 K 2023-04-07 01:55:17,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ff332f26550df115caee283ff76382bb in 854ms, sequenceid=950, compaction requested=false 2023-04-07 01:55:17,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1a6cfb3c84c64444bf3bca3b65c0b564, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1fed4a21871a45fb9c05baf9cb101094, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7a0e7aa862446a0b737d553935e3d55, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ede9ecc6f92b4b4d816e2eeb79dc2059, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ce0950768318401d89b96f6c28a4bdcf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7d97a6d7e08437c8eec2e0dc4011fca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bdc51b021e0c48f2a02b6f4eb725c34b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3cf12ac148cf434b8546912f407967b3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/6593c79589f1477680fc9149e7e5d0a4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bd13c657d26a4b278bea3371ba3a944a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463] to archive 2023-04-07 01:55:17,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:17,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/98b7b77f0a6848baa5282f94c0c6f45f 2023-04-07 01:55:17,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9c10a300f3394e77baf59b819c358f59 2023-04-07 01:55:17,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/2bd6f7d70eb743c280962eb3b298f4f1 2023-04-07 01:55:17,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/99c257c5e7944a4cab687c7e7917cbca 2023-04-07 01:55:17,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/894f693ded1b4774a450be8997edbd77 2023-04-07 01:55:17,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/817ce063d5374b61a2ea347cc1403e46 2023-04-07 01:55:17,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1a6cfb3c84c64444bf3bca3b65c0b564 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1a6cfb3c84c64444bf3bca3b65c0b564 2023-04-07 01:55:17,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/dc1ef85b4316408683e17f2ff9b2ebc0 2023-04-07 01:55:17,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3768bc73e0054b0e8db972bf041d0cb2 2023-04-07 01:55:17,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1fed4a21871a45fb9c05baf9cb101094 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/1fed4a21871a45fb9c05baf9cb101094 2023-04-07 01:55:17,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/86a844de2bb0486eb9bd06fcebb582f2 2023-04-07 01:55:17,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/446292a2e61041128a8812dcff01823f 2023-04-07 01:55:17,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7a0e7aa862446a0b737d553935e3d55 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7a0e7aa862446a0b737d553935e3d55 2023-04-07 01:55:17,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e394eb157fea45c993a6bb41595d0d67 2023-04-07 01:55:17,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/fab4d7929f364bccab0957a21271fe05 2023-04-07 01:55:17,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/53386f21806841f681876097bb4d78e6 2023-04-07 01:55:17,115 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/a8b87695bd8f4434bfb4ef19a7961085 2023-04-07 01:55:17,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/5ea576ea2dcb4d7da1c3b1961b39a305 2023-04-07 01:55:17,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ede9ecc6f92b4b4d816e2eeb79dc2059 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ede9ecc6f92b4b4d816e2eeb79dc2059 2023-04-07 01:55:17,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7c021b12a7e74edd8b8da032a4b5ba03 2023-04-07 01:55:17,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/312b6bbba68744a8a87d50306d3ba029 2023-04-07 01:55:17,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/15689a4d8f2f455fb27203dcb0fb8730 2023-04-07 01:55:17,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ce0950768318401d89b96f6c28a4bdcf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ce0950768318401d89b96f6c28a4bdcf 2023-04-07 01:55:17,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ef2c2c02de704fb78bba0c86233d9e0f 2023-04-07 01:55:17,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/b07c79c1584b4073917c76069d684e33 2023-04-07 01:55:17,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7d97a6d7e08437c8eec2e0dc4011fca to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7d97a6d7e08437c8eec2e0dc4011fca 2023-04-07 01:55:17,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/0ab31dbc87ce4b6ebcb5d20d7d901c7f 2023-04-07 01:55:17,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bdc51b021e0c48f2a02b6f4eb725c34b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bdc51b021e0c48f2a02b6f4eb725c34b 2023-04-07 01:55:17,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/57ea68d0ece1412c8f8c5188bf0e1b1b 2023-04-07 01:55:17,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/43c5e4a6fa3e40c39f02f30ac31ced8e 2023-04-07 01:55:17,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d608d51e722246c5b55ce652fca51506 2023-04-07 01:55:17,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3cf12ac148cf434b8546912f407967b3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/3cf12ac148cf434b8546912f407967b3 2023-04-07 01:55:17,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/81fe1ff34b6e4bd69b8fbdfe56a3abdc 2023-04-07 01:55:17,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/4b6c91d971ae40fa9e2b9dbe5490277a 2023-04-07 01:55:17,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/18724a3e8f794e9dbceb6a3fe11e8279 2023-04-07 01:55:17,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/9d3481ea61504507a570fa7f49242b95 2023-04-07 01:55:17,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/6593c79589f1477680fc9149e7e5d0a4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/6593c79589f1477680fc9149e7e5d0a4 2023-04-07 01:55:17,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/ae81799b07ab4c398729ff1eb8174b49 2023-04-07 01:55:17,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c2b37265b8214a9c8f21ac72e98d24bc 2023-04-07 01:55:17,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bd13c657d26a4b278bea3371ba3a944a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/bd13c657d26a4b278bea3371ba3a944a 2023-04-07 01:55:17,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/d436957ff4f748c28c49ec0ab026002d 2023-04-07 01:55:17,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/7226a2a3481e42c49fa197d6422cc463 2023-04-07 01:55:17,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/38599f3d8c104d9287e82835eea566f9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/95074500653d4f4180e9f9a80f350295, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a55a427388574f338c0dc582570112d6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/08808cfc51b542079f37c7080455d562, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5321e3fbc12c4daab65167ff1afa54de, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/dae637f7d95b4fc5b43ad017d08c9d2c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/554ab79be8404dea846d2124c301a762, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1008f213d1ca4a81b875e731dc1db0b4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/3fc2b562a4194644b47dfc41b3f2e01a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/931a03d7bed34bb1909abfc39e0421d5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04] to archive 2023-04-07 01:55:17,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:17,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bd5fcbf1f8654e82bced1edf56aa95fa 2023-04-07 01:55:17,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/318ffe69509741259d4e47bceefef4cf 2023-04-07 01:55:17,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d90aa4d8bb684c01a7a2b7381f53dff3 2023-04-07 01:55:17,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/890ce935ca0c46338d1eb81f8862c10d 2023-04-07 01:55:17,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/e0dc28e42fcf448abf7000d0cdf3823c 2023-04-07 01:55:17,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/38599f3d8c104d9287e82835eea566f9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/38599f3d8c104d9287e82835eea566f9 2023-04-07 01:55:17,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a7cba3789ed4d0a8ccc7ae33f2509c7 2023-04-07 01:55:17,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/93df8997845041819b5b885c89ec7895 2023-04-07 01:55:17,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/95074500653d4f4180e9f9a80f350295 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/95074500653d4f4180e9f9a80f350295 2023-04-07 01:55:17,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/f97386d4d8d140adac7ef3a140efe3e9 2023-04-07 01:55:17,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6d039e8c1d1f4f079886fb2c1c7e8f8a 2023-04-07 01:55:17,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15b0d202045744aa9144059dad2ee422 2023-04-07 01:55:17,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a55a427388574f338c0dc582570112d6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a55a427388574f338c0dc582570112d6 2023-04-07 01:55:17,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/b2ebed7388ce403094d06df8f14ad80f 2023-04-07 01:55:17,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/6b413e9eafcd4f5997258855b8804314 2023-04-07 01:55:17,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a643051482847cb8830a5fe59eb6362 2023-04-07 01:55:17,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/8e94c5240aaa4f719eb9355f13ecfed4 2023-04-07 01:55:17,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/08808cfc51b542079f37c7080455d562 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/08808cfc51b542079f37c7080455d562 2023-04-07 01:55:17,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/da36948101bd48449313f4a3f6031bbd 2023-04-07 01:55:17,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/d462679421184deba725a344acbec530 2023-04-07 01:55:17,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0495807ea70a4a479e426b4bfaefb8a1 2023-04-07 01:55:17,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5321e3fbc12c4daab65167ff1afa54de to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5321e3fbc12c4daab65167ff1afa54de 2023-04-07 01:55:17,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/09b3ce5ed80b4d17a90034716836d217 2023-04-07 01:55:17,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fa6886591bfd4e3dbff40f00044c7882 2023-04-07 01:55:17,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/dae637f7d95b4fc5b43ad017d08c9d2c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/dae637f7d95b4fc5b43ad017d08c9d2c 2023-04-07 01:55:17,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bbae6fc220e74115a2efdcf8cb740c8a 2023-04-07 01:55:17,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/30fcbd7c83cd4ed0bc411ec80f41eaca 2023-04-07 01:55:17,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:17,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/554ab79be8404dea846d2124c301a762 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/554ab79be8404dea846d2124c301a762 2023-04-07 01:55:17,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/fcda182b4d3c413586690b54a2f1a5d2 2023-04-07 01:55:17,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/bf0bd9327f4049fdaa3d238ad50a001a 2023-04-07 01:55:17,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0818b903090d4500bddc8daba5f6b4b8 2023-04-07 01:55:17,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1008f213d1ca4a81b875e731dc1db0b4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1008f213d1ca4a81b875e731dc1db0b4 2023-04-07 01:55:17,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/255862264f95479fa10ae50d5287e19a 2023-04-07 01:55:17,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/1b2fa4d24aad45dfb98ab0d5a0fb6687 2023-04-07 01:55:17,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/0a3d3467181f441c8315784ed8afb013 2023-04-07 01:55:17,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/3fc2b562a4194644b47dfc41b3f2e01a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/3fc2b562a4194644b47dfc41b3f2e01a 2023-04-07 01:55:17,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6f88136785e490282bb31e20bee475f 2023-04-07 01:55:17,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c6e09807c2e2463ebce31b52f2fd014d 2023-04-07 01:55:17,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/931a03d7bed34bb1909abfc39e0421d5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/931a03d7bed34bb1909abfc39e0421d5 2023-04-07 01:55:17,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/a77100f7468048bbbbe70521319b7b6b 2023-04-07 01:55:17,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/4fd27c6599ad4f4799f7756bd1d44aac 2023-04-07 01:55:17,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/5a88f06a52ff4b5ebd7144e066c4ff04 2023-04-07 01:55:17,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94f46c13302744549f34ef44d65acd56, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c40efa72c4a1479a90293113c4edfe86, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/cc49b46d1663475da1c3e8185b3b49cc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6e3f74917b0c4340ae4601bd69f76f0a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b940a79086da46db8d9cb212c860bc3f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6785efa7863c4a0c84ad0a6d9599d9a0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6bf4602822f34dceab30557b25cb3337, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b46d0701dcf44126beb5c3a2fb708adb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d24c9047ee384abca4cf2e55259a2484, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d] to archive 2023-04-07 01:55:17,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:17,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/37ad0f9019b4426fa7647d8dd3aa4309 2023-04-07 01:55:17,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/26652907d8c4435598979dc58d82caf8 2023-04-07 01:55:17,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964cc1178399463fb88c98152e38da33 2023-04-07 01:55:17,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/411bc8541bdd4330b6c97a747dad9420 2023-04-07 01:55:17,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/497e46c1ccd641259b1187d2bd72e8c0 2023-04-07 01:55:17,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94f46c13302744549f34ef44d65acd56 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94f46c13302744549f34ef44d65acd56 2023-04-07 01:55:17,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1e64dc83b95148748d40c6d7e9bfbf00 2023-04-07 01:55:17,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/e92359741030405db4de45b27981a8f1 2023-04-07 01:55:17,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c40efa72c4a1479a90293113c4edfe86 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c40efa72c4a1479a90293113c4edfe86 2023-04-07 01:55:17,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/76ee3b18012c4f67a35565fbe8be3aa5 2023-04-07 01:55:17,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a8ef250e2a544dff9c63ddde9d78aaac 2023-04-07 01:55:17,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c35e5f30f4a24e71becf6d3008333f47 2023-04-07 01:55:17,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/1256421390554f418c4bd672a5a9fbaf 2023-04-07 01:55:17,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/cc49b46d1663475da1c3e8185b3b49cc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/cc49b46d1663475da1c3e8185b3b49cc 2023-04-07 01:55:17,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/23b0ec4919df4b3b97db7915c9a6c4fc 2023-04-07 01:55:17,198 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0ec103e37c044791af5b6d5ee318b155 2023-04-07 01:55:17,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/62b8050c4ea447e8b1a77c301006956d 2023-04-07 01:55:17,200 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6e3f74917b0c4340ae4601bd69f76f0a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6e3f74917b0c4340ae4601bd69f76f0a 2023-04-07 01:55:17,201 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/892ce032143f47a2bf1d6e24cca4a7a7 2023-04-07 01:55:17,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/780446d018d64502a46720af8f7da8d9 2023-04-07 01:55:17,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/964c6b7de04e402b8128db0a0db36295 2023-04-07 01:55:17,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/94a60e63dd1d40d2a2f6456e2a9e3302 2023-04-07 01:55:17,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b940a79086da46db8d9cb212c860bc3f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b940a79086da46db8d9cb212c860bc3f 2023-04-07 01:55:17,207 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d510368af35c49f5bdb578c99513f68b 2023-04-07 01:55:17,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/52dffc8858014b12a6521a0e4b743cae 2023-04-07 01:55:17,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/79cefa46c85a48549483ba028d9a774f 2023-04-07 01:55:17,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6785efa7863c4a0c84ad0a6d9599d9a0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6785efa7863c4a0c84ad0a6d9599d9a0 2023-04-07 01:55:17,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/a7e7e42f8c6d48d8879a8f54098391e2 2023-04-07 01:55:17,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6a5bc992068947728fa1f91b58000ff3 2023-04-07 01:55:17,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/4f2aa99e6e3c468cb7a8aacfa4405523 2023-04-07 01:55:17,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6bf4602822f34dceab30557b25cb3337 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/6bf4602822f34dceab30557b25cb3337 2023-04-07 01:55:17,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/02bb8715c060418e8d665a27fa82fef3 2023-04-07 01:55:17,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b0c9d2307080474c9e5a1f70bde503a0 2023-04-07 01:55:17,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/c69385f2294f4fd7bf3f49d2b9cb4b59 2023-04-07 01:55:17,216 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b46d0701dcf44126beb5c3a2fb708adb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/b46d0701dcf44126beb5c3a2fb708adb 2023-04-07 01:55:17,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/fedc1cb04f6e4694bcffbebfeb753c66 2023-04-07 01:55:17,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/40b47830b85e46e89273914ae5c89ea1 2023-04-07 01:55:17,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d24c9047ee384abca4cf2e55259a2484 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/d24c9047ee384abca4cf2e55259a2484 2023-04-07 01:55:17,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/71ac76bafe6a42189950a4ab8dafb659 2023-04-07 01:55:17,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0f8deab68d184cbabb546d6e9111346b 2023-04-07 01:55:17,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/19b1f903023b472ab1b28664af5b867d 2023-04-07 01:55:17,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/recovered.edits/953.seqid, newMaxSeqId=953, maxSeqId=1 2023-04-07 01:55:17,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb. 2023-04-07 01:55:17,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ff332f26550df115caee283ff76382bb: 2023-04-07 01:55:17,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ff332f26550df115caee283ff76382bb 2023-04-07 01:55:17,227 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ff332f26550df115caee283ff76382bb, regionState=CLOSED 2023-04-07 01:55:17,227 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832517227"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832517227"}]},"ts":"1680832517227"} 2023-04-07 01:55:17,229 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-07 01:55:17,229 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure ff332f26550df115caee283ff76382bb, server=jenkins-hbase4.apache.org,44961,1680832421898 in 1.1600 sec 2023-04-07 01:55:17,230 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-07 01:55:17,230 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ff332f26550df115caee283ff76382bb, UNASSIGN in 1.1640 sec 2023-04-07 01:55:17,230 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832517230"}]},"ts":"1680832517230"} 2023-04-07 01:55:17,231 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:55:17,232 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:55:17,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-07 01:55:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 01:55:18,166 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-07 01:55:18,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:55:18,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,169 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-07 01:55:18,169 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,173 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:55:18,175 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/recovered.edits] 2023-04-07 01:55:18,178 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7331c2d6dc547f7b451ed27ea95bcd0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/c7331c2d6dc547f7b451ed27ea95bcd0 2023-04-07 01:55:18,179 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e4120e64059f4d3697d965f4d7ea824c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/A/e4120e64059f4d3697d965f4d7ea824c 2023-04-07 01:55:18,181 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15d5298ce93246f48203b2333158a518 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/15d5298ce93246f48203b2333158a518 2023-04-07 01:55:18,182 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c8258148468c490284093184f93b9405 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/B/c8258148468c490284093184f93b9405 2023-04-07 01:55:18,185 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0678c63c48944aa5b7a912e17727be49 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/0678c63c48944aa5b7a912e17727be49 2023-04-07 01:55:18,186 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/87ee25557eb648ff8b069a5c0c58b0ba to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/C/87ee25557eb648ff8b069a5c0c58b0ba 2023-04-07 01:55:18,188 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/recovered.edits/953.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb/recovered.edits/953.seqid 2023-04-07 01:55:18,189 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/ff332f26550df115caee283ff76382bb 2023-04-07 01:55:18,189 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:55:18,191 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,192 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:55:18,194 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:55:18,195 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,195 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:55:18,195 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832518195"}]},"ts":"9223372036854775807"} 2023-04-07 01:55:18,196 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:55:18,196 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ff332f26550df115caee283ff76382bb, NAME => 'TestAcidGuarantees,,1680832486189.ff332f26550df115caee283ff76382bb.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:55:18,197 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:55:18,197 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832518197"}]},"ts":"9223372036854775807"} 2023-04-07 01:55:18,198 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:55:18,200 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,200 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-07 01:55:18,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-07 01:55:18,270 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-07 01:55:18,281 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 226), OpenFileDescriptor=539 (was 520) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=325 (was 371), ProcessCount=169 (was 169), AvailableMemoryMB=2549 (was 2738) 2023-04-07 01:55:18,292 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=169, AvailableMemoryMB=2548 2023-04-07 01:55:18,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:55:18,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:55:18,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:18,296 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:55:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-07 01:55:18,297 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:55:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:18,298 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:18,298 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec empty. 2023-04-07 01:55:18,298 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:18,298 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:55:18,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:18,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:18,705 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:55:18,707 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 16a7d0fcb00e6e5b8493de38ebf26fec, NAME => 'TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:55:18,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:19,112 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:55:19,113 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 16a7d0fcb00e6e5b8493de38ebf26fec, disabling compactions & flushes 2023-04-07 01:55:19,113 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,113 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,113 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. after waiting 0 ms 2023-04-07 01:55:19,113 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,113 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,113 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:19,115 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:55:19,116 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832519116"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832519116"}]},"ts":"1680832519116"} 2023-04-07 01:55:19,117 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:55:19,118 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:55:19,118 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832519118"}]},"ts":"1680832519118"} 2023-04-07 01:55:19,119 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:55:19,122 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, ASSIGN}] 2023-04-07 01:55:19,124 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, ASSIGN 2023-04-07 01:55:19,125 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:55:19,276 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:19,276 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832519276"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832519276"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832519276"}]},"ts":"1680832519276"} 2023-04-07 01:55:19,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:19,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 16a7d0fcb00e6e5b8493de38ebf26fec, NAME => 'TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:55:19,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:55:19,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,433 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,434 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A 2023-04-07 01:55:19,434 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A 2023-04-07 01:55:19,435 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:19,435 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName A 2023-04-07 01:55:19,435 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:19,435 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,436 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B 2023-04-07 01:55:19,437 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B 2023-04-07 01:55:19,437 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:19,437 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName B 2023-04-07 01:55:19,437 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:19,438 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,438 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C 2023-04-07 01:55:19,439 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C 2023-04-07 01:55:19,439 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:19,439 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName C 2023-04-07 01:55:19,439 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:19,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:55:19,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:19,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:55:19,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 16a7d0fcb00e6e5b8493de38ebf26fec; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72669932, jitterRate=0.082866370677948}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:55:19,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:19,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., pid=37, masterSystemTime=1680832519429 2023-04-07 01:55:19,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:19,447 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:19,447 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832519447"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832519447"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832519447"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832519447"}]},"ts":"1680832519447"} 2023-04-07 01:55:19,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-07 01:55:19,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 in 171 msec 2023-04-07 01:55:19,450 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-07 01:55:19,450 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, ASSIGN in 327 msec 2023-04-07 01:55:19,451 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:55:19,451 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832519451"}]},"ts":"1680832519451"} 2023-04-07 01:55:19,452 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:55:19,453 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:55:19,454 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-07 01:55:20,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 01:55:20,401 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-07 01:55:20,402 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:20,407 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:20,407 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:20,409 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:20,410 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:55:20,411 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:55:20,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:55:20,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:55:20,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:20,822 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:55:20,823 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:55:20,823 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:55:20,825 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 01:55:20,827 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, REOPEN/MOVE}] 2023-04-07 01:55:20,828 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, REOPEN/MOVE 2023-04-07 01:55:20,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:20,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832520828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832520828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832520828"}]},"ts":"1680832520828"} 2023-04-07 01:55:20,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:20,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:20,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 16a7d0fcb00e6e5b8493de38ebf26fec, disabling compactions & flushes 2023-04-07 01:55:20,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:20,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:20,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. after waiting 0 ms 2023-04-07 01:55:20,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:20,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 01:55:20,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:20,985 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 16a7d0fcb00e6e5b8493de38ebf26fec to self. 2023-04-07 01:55:20,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:20,987 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=CLOSED 2023-04-07 01:55:20,987 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832520987"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832520987"}]},"ts":"1680832520987"} 2023-04-07 01:55:20,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-07 01:55:20,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 in 159 msec 2023-04-07 01:55:20,989 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=true 2023-04-07 01:55:21,139 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,140 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832521139"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832521139"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832521139"}]},"ts":"1680832521139"} 2023-04-07 01:55:21,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:21,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 16a7d0fcb00e6e5b8493de38ebf26fec, NAME => 'TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:55:21,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:55:21,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,297 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,298 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A 2023-04-07 01:55:21,298 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A 2023-04-07 01:55:21,298 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:21,298 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName A 2023-04-07 01:55:21,299 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:21,300 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,300 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B 2023-04-07 01:55:21,300 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B 2023-04-07 01:55:21,301 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:21,301 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName B 2023-04-07 01:55:21,301 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:21,301 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,302 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C 2023-04-07 01:55:21,302 DEBUG [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C 2023-04-07 01:55:21,302 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:21,302 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a7d0fcb00e6e5b8493de38ebf26fec columnFamilyName C 2023-04-07 01:55:21,302 INFO [StoreOpener-16a7d0fcb00e6e5b8493de38ebf26fec-1] regionserver.HStore(310): Store=16a7d0fcb00e6e5b8493de38ebf26fec/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:21,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:55:21,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 16a7d0fcb00e6e5b8493de38ebf26fec; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59298930, jitterRate=-0.1163770854473114}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:55:21,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:21,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., pid=42, masterSystemTime=1680832521292 2023-04-07 01:55:21,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,308 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,309 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832521308"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832521308"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832521308"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832521308"}]},"ts":"1680832521308"} 2023-04-07 01:55:21,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-07 01:55:21,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 in 168 msec 2023-04-07 01:55:21,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-07 01:55:21,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, REOPEN/MOVE in 484 msec 2023-04-07 01:55:21,313 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-07 01:55:21,313 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-07 01:55:21,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-07 01:55:21,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-07 01:55:21,316 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,320 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,320 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,325 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,325 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,328 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,329 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,332 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,332 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,335 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,336 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,340 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,340 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,344 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,344 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,349 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,349 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,353 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,353 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:21,358 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:21,363 DEBUG [hconnection-0x2b6c2464-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,364 DEBUG [hconnection-0x373f819d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:21,364 DEBUG [hconnection-0x9e976a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,365 DEBUG [hconnection-0x7fde0c45-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,365 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,366 DEBUG [hconnection-0x13b11865-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,366 DEBUG [hconnection-0x65fe9eb6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,367 DEBUG [hconnection-0x631d76b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,367 DEBUG [hconnection-0x72ee7081-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,367 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,367 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,367 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,367 DEBUG [hconnection-0x26570df0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,370 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:21,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:21,370 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:21,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:21,370 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,372 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,370 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:21,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:21,371 DEBUG [hconnection-0x640b634a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:21,372 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:21,375 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:21,375 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:21,375 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:21,376 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,376 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:21,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38468, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:21,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:21,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:21,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:21,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:21,395 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:55:21,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:21,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:21,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:21,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:21,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:21,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:21,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:21,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832581412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832581411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832581412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832581413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832581414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,420 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407ca6990375b6b468b9af054c6da06af50 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ca6990375b6b468b9af054c6da06af50 2023-04-07 01:55:21,427 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/e60c2cba74d94ef4843a6af911d66100 2023-04-07 01:55:21,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832581515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832581515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832581515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832581515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832581515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832581720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832581721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832581721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832581728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:21,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832581729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,842 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/56280fdd1f2b46faaa6b242c54be04d1 2023-04-07 01:55:21,857 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/f227e39896dd47cdb82937b200aa2b29 2023-04-07 01:55:21,862 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/e60c2cba74d94ef4843a6af911d66100 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100 2023-04-07 01:55:21,865 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100, entries=150, sequenceid=17, filesize=25.1 K 2023-04-07 01:55:21,866 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/56280fdd1f2b46faaa6b242c54be04d1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1 2023-04-07 01:55:21,869 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1, entries=150, sequenceid=17, filesize=11.6 K 2023-04-07 01:55:21,870 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/f227e39896dd47cdb82937b200aa2b29 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29 2023-04-07 01:55:21,875 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29, entries=150, sequenceid=17, filesize=11.6 K 2023-04-07 01:55:21,876 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a7d0fcb00e6e5b8493de38ebf26fec in 481ms, sequenceid=17, compaction requested=false 2023-04-07 01:55:21,876 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:21,876 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:21,878 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:21,879 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,879 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356e8c97[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:21,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:21,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:21,882 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:21,885 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:21,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:21,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:21,892 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:21,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:21,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:21,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:21,892 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:21,894 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,894 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:21,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,894 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:21,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:21,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:21,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:21,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:21,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,900 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:21,900 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:21,900 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:21,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:21,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:21,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:22,000 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:22,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:22,004 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:22,004 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:22,004 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:22,004 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:22,004 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,006 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:22,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:22,006 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,006 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:22,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:22,008 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:22,008 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:22,008 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:22,008 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:22,022 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407da49c76d086a4272a784c0077973aa4b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da49c76d086a4272a784c0077973aa4b 2023-04-07 01:55:22,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:22,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:22,033 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/fe00839e55d446ae91f2e1c8f96d7dc2 2023-04-07 01:55:22,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832582056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832582057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832582058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832582058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832582059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832582164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832582164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832582164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832582165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832582165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832582368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832582369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832582369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832582369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832582369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,441 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ae46b7620bda403fb873abcf63c4f7e0 2023-04-07 01:55:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832582673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832582673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832582673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832582674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:22,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832582675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,751 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:55:22,852 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/593525f874234ec5bba42db604db5b99 2023-04-07 01:55:22,856 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/fe00839e55d446ae91f2e1c8f96d7dc2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2 2023-04-07 01:55:22,859 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2, entries=150, sequenceid=36, filesize=25.1 K 2023-04-07 01:55:22,859 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ae46b7620bda403fb873abcf63c4f7e0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0 2023-04-07 01:55:22,862 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0, entries=150, sequenceid=36, filesize=11.6 K 2023-04-07 01:55:22,863 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/593525f874234ec5bba42db604db5b99 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99 2023-04-07 01:55:22,867 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99, entries=150, sequenceid=36, filesize=11.6 K 2023-04-07 01:55:22,868 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 16a7d0fcb00e6e5b8493de38ebf26fec in 860ms, sequenceid=36, compaction requested=false 2023-04-07 01:55:22,868 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:22,868 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,870 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:22,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:22,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3114254a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:22,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:22,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:22,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:22,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:22,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:22,879 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:22,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:22,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:22,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:22,879 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:22,882 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:22,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,892 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:22,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:22,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:22,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:22,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:22,893 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:22,893 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:22,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:22,993 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:22,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:22,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:22,997 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:22,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:22,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:22,999 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:22,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:23,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:23,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:23,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:23,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:23,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:23,002 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:23,002 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:23,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:23,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:23,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832583195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832583200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832583201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832583202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832583202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832583303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832583303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832583310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832583310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832583310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,422 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304073a1a7981660943b7b24b7e0a0a18b6fb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073a1a7981660943b7b24b7e0a0a18b6fb 2023-04-07 01:55:23,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832583507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832583507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832583513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832583513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832583514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832583811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832583813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832583819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832583819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832583820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:23,828 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/9663c672bd1943b483881b30fbc8a704 2023-04-07 01:55:24,238 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/b37c6ed2d03748bdb350deb74abf5647 2023-04-07 01:55:24,249 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/e83232c3427c4afca4e96bbb1e3815d3 2023-04-07 01:55:24,253 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/9663c672bd1943b483881b30fbc8a704 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704 2023-04-07 01:55:24,257 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704, entries=150, sequenceid=49, filesize=25.1 K 2023-04-07 01:55:24,258 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/b37c6ed2d03748bdb350deb74abf5647 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647 2023-04-07 01:55:24,261 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647, entries=150, sequenceid=49, filesize=11.6 K 2023-04-07 01:55:24,261 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/e83232c3427c4afca4e96bbb1e3815d3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3 2023-04-07 01:55:24,266 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3, entries=150, sequenceid=49, filesize=11.6 K 2023-04-07 01:55:24,266 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1264ms, sequenceid=49, compaction requested=true 2023-04-07 01:55:24,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:24,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,269 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445bcaee[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,278 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,278 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,285 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,285 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,285 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,285 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,285 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,285 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,285 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,285 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:24,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:55:24,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:24,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:24,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:24,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:24,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:24,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:24,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832584333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832584334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832584335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832584336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832584336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,386 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:24,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:24,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:24,389 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:24,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:24,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:24,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:24,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:24,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:24,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:24,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:24,393 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:24,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:24,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,395 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e8ccc9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,401 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,404 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,404 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,407 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:24,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,410 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,410 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832584441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832584441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832584442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832584442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832584446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,510 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:24,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:24,513 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:24,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:24,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:24,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:24,517 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,517 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:24,517 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:24,517 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,520 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3e4f7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,523 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,523 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,524 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,527 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,527 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,527 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,529 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,534 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,535 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,635 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:24,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:24,639 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:24,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:24,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:24,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,640 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:24,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:24,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:24,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832584646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832584646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832584647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832584647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650db17c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,655 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,657 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832584652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,659 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,659 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,662 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:24,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,665 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,665 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,735 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304077b37f39d92df4757bba5273142bc2d73 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077b37f39d92df4757bba5273142bc2d73 2023-04-07 01:55:24,741 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/331a9d79358a43c2878f1502e8b6b51d 2023-04-07 01:55:24,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/dc45c2d86e8f4ea5802e11dcc6c2101f 2023-04-07 01:55:24,765 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:24,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:24,769 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:24,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:24,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:24,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,771 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:24,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:24,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:24,773 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,774 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:24,774 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:24,774 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9b0b89[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,780 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,782 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,782 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,785 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,787 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,790 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,791 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,791 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,891 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:24,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:24,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:24,895 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:24,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:24,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:24,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:24,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:24,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:24,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:24,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:24,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:24,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:24,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:24,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@de71e37[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:24,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:24,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:24,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:24,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:24,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:24,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:24,908 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:24,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:24,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:24,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:24,908 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:24,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:24,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:24,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:24,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:24,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:24,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:24,914 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:24,914 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:24,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:24,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832584951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832584952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832584952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832584953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:24,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:24,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832584959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,014 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:25,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:25,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:25,018 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:25,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:25,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:25,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:25,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:25,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:25,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:25,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:25,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:25,029 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,029 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:25,029 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:25,029 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:25,033 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:25,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:25,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6789c720[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:25,035 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:25,035 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,036 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:25,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,036 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:25,039 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:25,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:25,042 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:25,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:25,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:25,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:25,042 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:25,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:25,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:25,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:25,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:25,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:25,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:25,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,050 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:25,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:25,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,050 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:25,151 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:25,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:25,154 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:25,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:25,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:25,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,156 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:25,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:25,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:25,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:25,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:25,157 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,157 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:25,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:25,158 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:25,160 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,160 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:25,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0893bd2e19b3471db4b5c640e849cd5c 2023-04-07 01:55:25,161 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,161 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:25,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f070cfc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:25,163 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:25,163 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,164 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:25,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:25,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:25,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:25,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/331a9d79358a43c2878f1502e8b6b51d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d 2023-04-07 01:55:25,166 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:25,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:25,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:25,167 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:25,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:25,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:25,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:25,167 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:25,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:25,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:25,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:25,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d, entries=200, sequenceid=69, filesize=31.9 K 2023-04-07 01:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:25,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/dc45c2d86e8f4ea5802e11dcc6c2101f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f 2023-04-07 01:55:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f, entries=150, sequenceid=69, filesize=11.6 K 2023-04-07 01:55:25,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0893bd2e19b3471db4b5c640e849cd5c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:25,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:25,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:25,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:25,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:25,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:25,176 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:25,176 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:25,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c, entries=150, sequenceid=69, filesize=11.6 K 2023-04-07 01:55:25,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 16a7d0fcb00e6e5b8493de38ebf26fec in 855ms, sequenceid=69, compaction requested=true 2023-04-07 01:55:25,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:25,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:25,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:25,179 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:25,179 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:25,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:25,180 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:25,180 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/B is initiating minor compaction (all files) 2023-04-07 01:55:25,181 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/B in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,181 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=46.4 K 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 56280fdd1f2b46faaa6b242c54be04d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832521393 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/A is initiating minor compaction (all files) 2023-04-07 01:55:25,181 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/A in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,181 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=107.3 K 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ae46b7620bda403fb873abcf63c4f7e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832521410 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e60c2cba74d94ef4843a6af911d66100, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832521393 2023-04-07 01:55:25,181 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b37c6ed2d03748bdb350deb74abf5647, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832522023 2023-04-07 01:55:25,182 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting fe00839e55d446ae91f2e1c8f96d7dc2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832521410 2023-04-07 01:55:25,182 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dc45c2d86e8f4ea5802e11dcc6c2101f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523200 2023-04-07 01:55:25,182 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9663c672bd1943b483881b30fbc8a704, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832522023 2023-04-07 01:55:25,188 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 331a9d79358a43c2878f1502e8b6b51d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523192 2023-04-07 01:55:25,198 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#B#compaction#350 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:25,217 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#A#compaction#351 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:25,222 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/e66f1fd2883247c3b50120794c27ed6a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e66f1fd2883247c3b50120794c27ed6a 2023-04-07 01:55:25,225 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/B of 16a7d0fcb00e6e5b8493de38ebf26fec into e66f1fd2883247c3b50120794c27ed6a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:25,226 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:25,226 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/B, priority=12, startTime=1680832525179; duration=0sec 2023-04-07 01:55:25,226 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:25,226 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:55:25,227 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:55:25,227 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/C is initiating minor compaction (all files) 2023-04-07 01:55:25,227 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/C in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,227 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=46.4 K 2023-04-07 01:55:25,229 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f227e39896dd47cdb82937b200aa2b29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832521393 2023-04-07 01:55:25,229 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 593525f874234ec5bba42db604db5b99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832521410 2023-04-07 01:55:25,230 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e83232c3427c4afca4e96bbb1e3815d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832522023 2023-04-07 01:55:25,230 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0893bd2e19b3471db4b5c640e849cd5c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523200 2023-04-07 01:55:25,245 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#C#compaction#352 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:25,257 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/58764c13c3224da3a342aee5dc26f4ef as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/58764c13c3224da3a342aee5dc26f4ef 2023-04-07 01:55:25,261 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/C of 16a7d0fcb00e6e5b8493de38ebf26fec into 58764c13c3224da3a342aee5dc26f4ef(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:25,261 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:25,261 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/C, priority=12, startTime=1680832525179; duration=0sec 2023-04-07 01:55:25,261 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:25,276 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:25,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:25,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:25,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:25,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:25,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:25,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:25,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:25,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:25,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:25,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:25,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:25,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:25,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:25,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:25,284 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:25,284 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:25,285 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:25,285 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:25,285 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:25,285 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:25,285 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:25,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:25,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832585479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832585480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832585481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832585482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832585483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832585584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832585588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832585590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832585590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832585590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,702 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e56b9a9aa7f3465a9149de973d851509 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e56b9a9aa7f3465a9149de973d851509 2023-04-07 01:55:25,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832585789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832585794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832585797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832585798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:25,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832585799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,056 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/ae5de6e639cf41b498f9919af7de8f05 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/ae5de6e639cf41b498f9919af7de8f05 2023-04-07 01:55:26,060 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/A of 16a7d0fcb00e6e5b8493de38ebf26fec into ae5de6e639cf41b498f9919af7de8f05(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:26,060 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:26,060 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/A, priority=12, startTime=1680832525178; duration=0sec 2023-04-07 01:55:26,060 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:26,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832586094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832586102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832586103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832586103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832586103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,107 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/6a32a757f96948fcb0e67123d5197cdd 2023-04-07 01:55:26,517 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/6c10a11e8b5f4b7da756c36f0acc0322 2023-04-07 01:55:26,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832586599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38456 deadline: 1680832586606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38434 deadline: 1680832586608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832586608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:26,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38396 deadline: 1680832586609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,927 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/ff27af94661241459b0ff14e3318a504 2023-04-07 01:55:26,930 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/6a32a757f96948fcb0e67123d5197cdd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd 2023-04-07 01:55:26,937 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd, entries=150, sequenceid=83, filesize=25.1 K 2023-04-07 01:55:26,937 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/6c10a11e8b5f4b7da756c36f0acc0322 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322 2023-04-07 01:55:26,941 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322, entries=150, sequenceid=83, filesize=11.6 K 2023-04-07 01:55:26,941 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/ff27af94661241459b0ff14e3318a504 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504 2023-04-07 01:55:26,945 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504, entries=150, sequenceid=83, filesize=11.6 K 2023-04-07 01:55:26,946 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1661ms, sequenceid=83, compaction requested=false 2023-04-07 01:55:26,946 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:26,946 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,947 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e795d6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:26,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:26,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:26,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:26,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:26,955 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:26,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:26,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:26,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:26,955 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:26,958 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,958 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,958 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:26,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:26,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:26,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:26,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:26,963 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:26,963 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:26,963 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:26,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:26,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,063 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:27,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:27,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:27,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:27,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:27,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:27,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:27,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:27,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,068 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,068 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:27,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:27,070 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:27,071 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,083 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304075488798dcebb4c4199c55d57811cb949 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075488798dcebb4c4199c55d57811cb949 2023-04-07 01:55:27,094 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=104, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/eef435bdba8c4c9d807becbeadd07f5d 2023-04-07 01:55:27,104 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ecf5f724d8024c328fee27e23d2f90c9 2023-04-07 01:55:27,113 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/74dd6c69b080447eb62af02102d1f86b 2023-04-07 01:55:27,118 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/eef435bdba8c4c9d807becbeadd07f5d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d 2023-04-07 01:55:27,121 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d, entries=150, sequenceid=104, filesize=25.1 K 2023-04-07 01:55:27,122 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ecf5f724d8024c328fee27e23d2f90c9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9 2023-04-07 01:55:27,125 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9, entries=150, sequenceid=104, filesize=11.6 K 2023-04-07 01:55:27,126 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/74dd6c69b080447eb62af02102d1f86b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b 2023-04-07 01:55:27,130 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b, entries=150, sequenceid=104, filesize=11.6 K 2023-04-07 01:55:27,131 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 16a7d0fcb00e6e5b8493de38ebf26fec in 60ms, sequenceid=104, compaction requested=true 2023-04-07 01:55:27,131 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:27,131 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:27,134 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc43c5b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:27,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:27,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:27,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:27,142 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:27,145 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:27,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:27,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:27,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:27,146 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:27,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,150 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,153 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:27,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,154 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:27,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,154 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:27,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:27,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,254 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:27,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:27,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:27,259 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:27,259 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,259 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:27,259 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:27,260 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,260 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,263 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:27,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:27,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:27,267 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,269 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:27,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54dd5404[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:27,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:27,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,272 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,273 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:27,278 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:27,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:27,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:27,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:27,279 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:27,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,288 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:27,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:27,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:27,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,289 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:27,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,289 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:27,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,389 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:27,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:27,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:27,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:27,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:27,394 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:27,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:27,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:27,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,395 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,395 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:27,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:27,400 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,400 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:27,400 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:27,401 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,402 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad28f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:27,405 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:27,405 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:27,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:27,411 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:27,411 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:27,411 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:27,411 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:27,411 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:27,412 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:27,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:27,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:27,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,418 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:27,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,418 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:27,518 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:27,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:27,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:27,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:27,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:27,524 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:27,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:27,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:27,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,525 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:27,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:27,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:27,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:27,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:27,528 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:27,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d7ae11[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:27,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:27,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,533 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:27,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:27,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:27,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:27,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:27,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:27,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:27,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:27,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:27,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:27,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:27,545 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,546 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:27,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:27,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:27,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:27,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:27,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,552 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:27,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:27,552 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:27,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,652 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:27,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:27,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:27,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:27,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:27,656 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:27,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:27,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:27,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,658 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:27,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:27,658 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,658 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:27,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:27,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:27,660 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:27,660 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:27,660 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:27,660 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:27,660 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:27,661 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:27,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:27,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:27,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832527693, max timestamp=1680832527705Num uniques 859; 2023-04-07 01:55:27,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:27,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832527693, max timestamp=1680832527705Num uniques 859; 2023-04-07 01:55:27,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:27,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832527693, max timestamp=1680832527705Num uniques 859; 2023-04-07 01:55:27,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832527755, max timestamp=1680832527779Num uniques 1168; 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832527755, max timestamp=1680832527779Num uniques 1168; 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832527755, max timestamp=1680832527779Num uniques 1168; 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-07 01:55:27,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832527814, max timestamp=1680832527817Num uniques 1077; 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832527814, max timestamp=1680832527817Num uniques 1077; 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832527814, max timestamp=1680832527817Num uniques 1077; 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-07 01:55:27,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-07 01:55:27,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680832527846, max timestamp=1680832527856Num uniques 1036; 2023-04-07 01:55:27,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-07 01:55:27,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680832527846, max timestamp=1680832527856Num uniques 1036; 2023-04-07 01:55:27,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-07 01:55:27,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680832527846, max timestamp=1680832527856Num uniques 1036; 2023-04-07 01:55:27,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-07 01:55:27,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:27,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832587889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832527866, max timestamp=1680832527890Num uniques 1245; 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832527866, max timestamp=1680832527890Num uniques 1245; 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832527866, max timestamp=1680832527890Num uniques 1245; 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-07 01:55:27,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-07 01:55:27,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832527911, max timestamp=1680832527932Num uniques 1104; 2023-04-07 01:55:27,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38408 deadline: 1680832587927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832527911, max timestamp=1680832527932Num uniques 1104; 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680832527911, max timestamp=1680832527932Num uniques 1104; 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:55:27,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680832527958, max timestamp=1680832527983Num uniques 1113; 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680832527958, max timestamp=1680832527983Num uniques 1113; 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680832527958, max timestamp=1680832527983Num uniques 1113; 2023-04-07 01:55:27,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-07 01:55:28,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680832528010, max timestamp=1680832528025Num uniques 1122; 2023-04-07 01:55:28,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680832528010, max timestamp=1680832528025Num uniques 1122; 2023-04-07 01:55:28,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680832528010, max timestamp=1680832528025Num uniques 1122; 2023-04-07 01:55:28,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-07 01:55:28,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-07 01:55:28,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-07 01:55:28,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832528044, max timestamp=1680832528055Num uniques 1131; 2023-04-07 01:55:28,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 01:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832528044, max timestamp=1680832528055Num uniques 1131; 2023-04-07 01:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832528044, max timestamp=1680832528055Num uniques 1131; 2023-04-07 01:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 01:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832528068, max timestamp=1680832528081Num uniques 1090; 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832528068, max timestamp=1680832528081Num uniques 1090; 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680832528068, max timestamp=1680832528081Num uniques 1090; 2023-04-07 01:55:28,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-07 01:55:28,116 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040706f00a11cfe440adb73953a5df2d1262 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040706f00a11cfe440adb73953a5df2d1262 2023-04-07 01:55:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832528095, max timestamp=1680832528115Num uniques 1099; 2023-04-07 01:55:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832528095, max timestamp=1680832528115Num uniques 1099; 2023-04-07 01:55:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:55:28,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:55:28,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832528095, max timestamp=1680832528115Num uniques 1099; 2023-04-07 01:55:28,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832528128, max timestamp=1680832528147Num uniques 1058; 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832528128, max timestamp=1680832528147Num uniques 1058; 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680832528128, max timestamp=1680832528147Num uniques 1058; 2023-04-07 01:55:28,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-07 01:55:28,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832528163, max timestamp=1680832528173Num uniques 1067; 2023-04-07 01:55:28,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832528163, max timestamp=1680832528173Num uniques 1067; 2023-04-07 01:55:28,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:55:28,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:55:28,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680832528163, max timestamp=1680832528173Num uniques 1067; 2023-04-07 01:55:28,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680832528196, max timestamp=1680832528204Num uniques 1076; 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680832528196, max timestamp=1680832528204Num uniques 1076; 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680832528196, max timestamp=1680832528204Num uniques 1076; 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-07 01:55:28,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-07 01:55:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832528226, max timestamp=1680832528235Num uniques 1085; 2023-04-07 01:55:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-07 01:55:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832528226, max timestamp=1680832528235Num uniques 1085; 2023-04-07 01:55:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-07 01:55:28,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680832528226, max timestamp=1680832528235Num uniques 1085; 2023-04-07 01:55:28,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-07 01:55:28,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680832528250, max timestamp=1680832528264Num uniques 1094; 2023-04-07 01:55:28,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-07 01:55:28,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680832528250, max timestamp=1680832528264Num uniques 1094; 2023-04-07 01:55:28,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-07 01:55:28,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680832528250, max timestamp=1680832528264Num uniques 1094; 2023-04-07 01:55:28,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-07 01:55:28,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832528290, max timestamp=1680832528297Num uniques 1153; 2023-04-07 01:55:28,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832528290, max timestamp=1680832528297Num uniques 1153; 2023-04-07 01:55:28,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-07 01:55:28,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680832528290, max timestamp=1680832528297Num uniques 1153; 2023-04-07 01:55:28,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-07 01:55:28,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-07 01:55:28,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680832528309, max timestamp=1680832528322Num uniques 1112; 2023-04-07 01:55:28,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-07 01:55:28,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680832528309, max timestamp=1680832528322Num uniques 1112; 2023-04-07 01:55:28,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680832528309, max timestamp=1680832528322Num uniques 1112; 2023-04-07 01:55:28,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-07 01:55:28,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-07 01:55:28,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680832528344, max timestamp=1680832528358Num uniques 1071; 2023-04-07 01:55:28,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-07 01:55:28,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680832528344, max timestamp=1680832528358Num uniques 1071; 2023-04-07 01:55:28,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-07 01:55:28,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680832528344, max timestamp=1680832528358Num uniques 1071; 2023-04-07 01:55:28,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-07 01:55:28,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38468 deadline: 1680832588382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680832528379, max timestamp=1680832528393Num uniques 1130; 2023-04-07 01:55:28,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-07 01:55:28,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680832528379, max timestamp=1680832528393Num uniques 1130; 2023-04-07 01:55:28,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-07 01:55:28,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680832528379, max timestamp=1680832528393Num uniques 1130; 2023-04-07 01:55:28,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680832528411, max timestamp=1680832528419Num uniques 1089; 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680832528411, max timestamp=1680832528419Num uniques 1089; 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680832528411, max timestamp=1680832528419Num uniques 1089; 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-07 01:55:28,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-07 01:55:28,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680832528446, max timestamp=1680832528453Num uniques 1148; 2023-04-07 01:55:28,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=9068, suffix heap size=147360,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=198 2023-04-07 01:55:28,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680832528446, max timestamp=1680832528453Num uniques 1148; 2023-04-07 01:55:28,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=9068, suffix heap size=147360,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=198 2023-04-07 01:55:28,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680832528446, max timestamp=1680832528453Num uniques 1148; 2023-04-07 01:55:28,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=9068, suffix heap size=147360,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=198 2023-04-07 01:55:28,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680832528503, max timestamp=1680832528514Num uniques 1057; 2023-04-07 01:55:28,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-07 01:55:28,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680832528503, max timestamp=1680832528514Num uniques 1057; 2023-04-07 01:55:28,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-07 01:55:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680832528503, max timestamp=1680832528514Num uniques 1057; 2023-04-07 01:55:28,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-07 01:55:28,522 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/3594862267ca434db2211fc2712ef6e1 2023-04-07 01:55:28,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680832528538, max timestamp=1680832528547Num uniques 1166; 2023-04-07 01:55:28,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=9886, suffix heap size=149232,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=216 2023-04-07 01:55:28,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680832528538, max timestamp=1680832528547Num uniques 1166; 2023-04-07 01:55:28,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680832528538, max timestamp=1680832528547Num uniques 1166; 2023-04-07 01:55:28,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=9886, suffix heap size=149232,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=216 2023-04-07 01:55:28,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=9886, suffix heap size=149232,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=216 2023-04-07 01:55:28,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680832528571, max timestamp=1680832528581Num uniques 1075; 2023-04-07 01:55:28,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=12590, suffix heap size=139768,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=275 2023-04-07 01:55:28,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680832528571, max timestamp=1680832528581Num uniques 1075; 2023-04-07 01:55:28,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680832528571, max timestamp=1680832528581Num uniques 1075; 2023-04-07 01:55:28,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=12590, suffix heap size=139768,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=275 2023-04-07 01:55:28,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=12590, suffix heap size=139768,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=275 2023-04-07 01:55:28,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680832528598, max timestamp=1680832528609Num uniques 1134; 2023-04-07 01:55:28,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=15294, suffix heap size=145904,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=334 2023-04-07 01:55:28,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680832528598, max timestamp=1680832528609Num uniques 1134; 2023-04-07 01:55:28,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=15294, suffix heap size=145904,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=334 2023-04-07 01:55:28,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680832528598, max timestamp=1680832528609Num uniques 1134; 2023-04-07 01:55:28,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=15294, suffix heap size=145904,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=334 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1680832528628, max timestamp=1680832528638Num uniques 1193; 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1680832528629, max timestamp=1680832528638Num uniques 1193; 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54638, new segment data size=13418, suffix heap size=152040,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1193, new segment cells count=293 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54638, new segment data size=8838, suffix heap size=152040,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1193, new segment cells count=193 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1680832528629, max timestamp=1680832528638Num uniques 1193; 2023-04-07 01:55:28,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54638, new segment data size=8838, suffix heap size=152040,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1193, new segment cells count=193 2023-04-07 01:55:28,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832588668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832588669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680832528659, max timestamp=1680832528670Num uniques 1052; 2023-04-07 01:55:28,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680832528659, max timestamp=1680832528670Num uniques 1052; 2023-04-07 01:55:28,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=13830, suffix heap size=137376,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=302 2023-04-07 01:55:28,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680832528659, max timestamp=1680832528670Num uniques 1152; 2023-04-07 01:55:28,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=13830, suffix heap size=137376,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=302 2023-04-07 01:55:28,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=13830, suffix heap size=147776,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=302 2023-04-07 01:55:28,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832588704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680832528684, max timestamp=1680832528711Num uniques 1161; 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680832528684, max timestamp=1680832528711Num uniques 1161; 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1680832528684, max timestamp=1680832528711Num uniques 1161; 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=11946, suffix heap size=148712,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=261 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=11946, suffix heap size=148712,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=261 2023-04-07 01:55:28,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=11946, suffix heap size=148712,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=261 2023-04-07 01:55:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680832528751, max timestamp=1680832528788Num uniques 1120; 2023-04-07 01:55:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680832528751, max timestamp=1680832528788Num uniques 1120; 2023-04-07 01:55:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-07 01:55:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-07 01:55:28,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680832528751, max timestamp=1680832528788Num uniques 1120; 2023-04-07 01:55:28,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-07 01:55:28,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680832528810, max timestamp=1680832528819Num uniques 1129; 2023-04-07 01:55:28,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680832528810, max timestamp=1680832528819Num uniques 1129; 2023-04-07 01:55:28,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680832528810, max timestamp=1680832528819Num uniques 1129; 2023-04-07 01:55:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-07 01:55:28,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-07 01:55:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680832528842, max timestamp=1680832528853Num uniques 1088; 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680832528842, max timestamp=1680832528853Num uniques 1088; 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680832528842, max timestamp=1680832528853Num uniques 1088; 2023-04-07 01:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=347, cellSize=15892, totalHeapSize=36328, min timestamp=1680832528867, max timestamp=1680832528886Num uniques 1097; 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=15892, suffix heap size=142056,new segment heap size=36328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=347 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=347, cellSize=15892, totalHeapSize=36328, min timestamp=1680832528867, max timestamp=1680832528886Num uniques 1097; 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=347, cellSize=15892, totalHeapSize=36328, min timestamp=1680832528867, max timestamp=1680832528886Num uniques 1097; 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=15892, suffix heap size=142056,new segment heap size=36328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=347 2023-04-07 01:55:28,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=15892, suffix heap size=142056,new segment heap size=36328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=347 2023-04-07 01:55:28,931 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/d17d0f7d5aac4eec935f4d2a08cec152 2023-04-07 01:55:28,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832588911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832588909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11716, totalHeapSize=26864, min timestamp=1680832528896, max timestamp=1680832528922Num uniques 1206; 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11716, totalHeapSize=26864, min timestamp=1680832528896, max timestamp=1680832528922Num uniques 1206; 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55226, new segment data size=11716, suffix heap size=153392,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1206, new segment cells count=256 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11716, totalHeapSize=26864, min timestamp=1680832528896, max timestamp=1680832528922Num uniques 1206; 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55226, new segment data size=11716, suffix heap size=153392,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1206, new segment cells count=256 2023-04-07 01:55:28,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55226, new segment data size=11716, suffix heap size=153392,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1206, new segment cells count=256 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680832528971, max timestamp=1680832529000Num uniques 1115; 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=9840, suffix heap size=143928,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=215 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680832528971, max timestamp=1680832529000Num uniques 1115; 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=9840, suffix heap size=143928,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=215 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680832528971, max timestamp=1680832529000Num uniques 1115; 2023-04-07 01:55:29,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=9840, suffix heap size=143928,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=215 2023-04-07 01:55:29,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1680832529039, max timestamp=1680832529048Num uniques 1074; 2023-04-07 01:55:29,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=7964, suffix heap size=139664,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=174 2023-04-07 01:55:29,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1680832529039, max timestamp=1680832529048Num uniques 1074; 2023-04-07 01:55:29,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=7964, suffix heap size=139664,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=174 2023-04-07 01:55:29,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1680832529039, max timestamp=1680832529048Num uniques 1074; 2023-04-07 01:55:29,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=7964, suffix heap size=139664,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=174 2023-04-07 01:55:29,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=333, cellSize=15248, totalHeapSize=34872, min timestamp=1680832529060, max timestamp=1680832529075Num uniques 1033; 2023-04-07 01:55:29,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=15248, suffix heap size=135400,new segment heap size=34872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=333 2023-04-07 01:55:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=333, cellSize=15248, totalHeapSize=34872, min timestamp=1680832529060, max timestamp=1680832529075Num uniques 1033; 2023-04-07 01:55:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=15248, suffix heap size=135400,new segment heap size=34872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=333 2023-04-07 01:55:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=333, cellSize=15248, totalHeapSize=34872, min timestamp=1680832529060, max timestamp=1680832529075Num uniques 1033; 2023-04-07 01:55:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=15248, suffix heap size=135400,new segment heap size=34872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=333 2023-04-07 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=342, cellSize=15662, totalHeapSize=35808, min timestamp=1680832529093, max timestamp=1680832529106Num uniques 1192; 2023-04-07 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=342, cellSize=15662, totalHeapSize=35808, min timestamp=1680832529093, max timestamp=1680832529106Num uniques 1192; 2023-04-07 01:55:29,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54592, new segment data size=15662, suffix heap size=151936,new segment heap size=35808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1192, new segment cells count=342 2023-04-07 01:55:29,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=342, cellSize=15662, totalHeapSize=35808, min timestamp=1680832529093, max timestamp=1680832529106Num uniques 1192; 2023-04-07 01:55:29,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54592, new segment data size=15662, suffix heap size=151936,new segment heap size=35808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1192, new segment cells count=342 2023-04-07 01:55:29,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54592, new segment data size=15662, suffix heap size=151936,new segment heap size=35808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1192, new segment cells count=342 2023-04-07 01:55:29,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11486, totalHeapSize=26344, min timestamp=1680832529126, max timestamp=1680832529143Num uniques 1201; 2023-04-07 01:55:29,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11486, totalHeapSize=26344, min timestamp=1680832529126, max timestamp=1680832529143Num uniques 1201; 2023-04-07 01:55:29,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54996, new segment data size=11486, suffix heap size=152872,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1201, new segment cells count=251 2023-04-07 01:55:29,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54996, new segment data size=11486, suffix heap size=152872,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1201, new segment cells count=251 2023-04-07 01:55:29,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11486, totalHeapSize=26344, min timestamp=1680832529126, max timestamp=1680832529143Num uniques 1201; 2023-04-07 01:55:29,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54996, new segment data size=11486, suffix heap size=152872,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1201, new segment cells count=251 2023-04-07 01:55:29,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=310, cellSize=14190, totalHeapSize=32480, min timestamp=1680832529166, max timestamp=1680832529176Num uniques 1110; 2023-04-07 01:55:29,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=14190, suffix heap size=143408,new segment heap size=32480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=310 2023-04-07 01:55:29,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=310, cellSize=14190, totalHeapSize=32480, min timestamp=1680832529166, max timestamp=1680832529176Num uniques 1110; 2023-04-07 01:55:29,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=310, cellSize=14190, totalHeapSize=32480, min timestamp=1680832529166, max timestamp=1680832529176Num uniques 1110; 2023-04-07 01:55:29,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=14190, suffix heap size=143408,new segment heap size=32480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=310 2023-04-07 01:55:29,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=14190, suffix heap size=143408,new segment heap size=32480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=310 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680832529195, max timestamp=1680832529208Num uniques 1169; 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53534, new segment data size=12314, suffix heap size=149544,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1169, new segment cells count=269 2023-04-07 01:55:29,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680832529195, max timestamp=1680832529208Num uniques 1169; 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680832529195, max timestamp=1680832529208Num uniques 1169; 2023-04-07 01:55:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832589198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53534, new segment data size=12314, suffix heap size=149544,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1169, new segment cells count=269 2023-04-07 01:55:29,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53534, new segment data size=12314, suffix heap size=149544,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1169, new segment cells count=269 2023-04-07 01:55:29,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1680832529239, max timestamp=1680832529268Num uniques 1128; 2023-04-07 01:55:29,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1680832529239, max timestamp=1680832529268Num uniques 1128; 2023-04-07 01:55:29,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=12728, suffix heap size=145280,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=278 2023-04-07 01:55:29,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=12728, suffix heap size=145280,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=278 2023-04-07 01:55:29,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1680832529239, max timestamp=1680832529268Num uniques 1128; 2023-04-07 01:55:29,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=12728, suffix heap size=145280,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=278 2023-04-07 01:55:29,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680832529289, max timestamp=1680832529311Num uniques 1137; 2023-04-07 01:55:29,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=10852, suffix heap size=146216,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=237 2023-04-07 01:55:29,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680832529289, max timestamp=1680832529311Num uniques 1137; 2023-04-07 01:55:29,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=10852, suffix heap size=146216,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=237 2023-04-07 01:55:29,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680832529289, max timestamp=1680832529311Num uniques 1137; 2023-04-07 01:55:29,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=10852, suffix heap size=146216,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=237 2023-04-07 01:55:29,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=346, cellSize=15846, totalHeapSize=36224, min timestamp=1680832529322, max timestamp=1680832529339Num uniques 1096; 2023-04-07 01:55:29,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=346, cellSize=15846, totalHeapSize=36224, min timestamp=1680832529322, max timestamp=1680832529339Num uniques 1096; 2023-04-07 01:55:29,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=346, cellSize=15846, totalHeapSize=36224, min timestamp=1680832529322, max timestamp=1680832529339Num uniques 1096; 2023-04-07 01:55:29,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=15846, suffix heap size=141952,new segment heap size=36224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=346 2023-04-07 01:55:29,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=15846, suffix heap size=141952,new segment heap size=36224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=346 2023-04-07 01:55:29,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=15846, suffix heap size=141952,new segment heap size=36224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=346 2023-04-07 01:55:29,342 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/f9ae4bda74be40ffae4a283e225c2bad 2023-04-07 01:55:29,346 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/3594862267ca434db2211fc2712ef6e1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1 2023-04-07 01:55:29,350 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1, entries=150, sequenceid=113, filesize=25.1 K 2023-04-07 01:55:29,350 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/d17d0f7d5aac4eec935f4d2a08cec152 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152 2023-04-07 01:55:29,354 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152, entries=150, sequenceid=113, filesize=11.6 K 2023-04-07 01:55:29,354 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/f9ae4bda74be40ffae4a283e225c2bad as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad 2023-04-07 01:55:29,357 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad, entries=150, sequenceid=113, filesize=11.6 K 2023-04-07 01:55:29,358 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=107.34 KB/109920 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1698ms, sequenceid=113, compaction requested=true 2023-04-07 01:55:29,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:29,358 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:29,360 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:29,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:29,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=120.76 KB heapSize=327.61 KB 2023-04-07 01:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:29,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:29,364 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d6254[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:29,364 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,368 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:29,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:29,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:29,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:29,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:29,373 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:29,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:29,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:29,373 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:29,373 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:29,375 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,375 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,375 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,376 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,377 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304072e3a69d6c63344ad9e81dd3bb9f8027e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e3a69d6c63344ad9e81dd3bb9f8027e 2023-04-07 01:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,377 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,380 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,380 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,381 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:29,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,381 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:29,381 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:29,382 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=883, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/56ac42ee9405433484423ce40c54bc24 2023-04-07 01:55:29,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832589371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832589380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832589381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832589382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832589383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,481 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:29,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:29,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:29,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:29,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:29,499 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:29,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:29,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:29,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:29,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:29,503 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,503 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:29,503 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:29,503 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,505 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832589496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832589496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832589496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832589496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,506 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832589497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:29,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ee7e41[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:29,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,508 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:29,508 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,508 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:29,510 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:29,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:29,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:29,512 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:29,512 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:29,512 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:29,512 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:29,512 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:29,514 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,514 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,516 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:29,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:29,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,518 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:29,519 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:29,519 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:29,519 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,519 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,619 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:29,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:29,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:29,622 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,624 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:29,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:29,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:29,626 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,626 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:29,626 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:29,626 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:29,628 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:29,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c528d9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:29,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:29,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,632 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:29,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:29,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:29,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:29,633 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:29,636 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:29,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:29,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:29,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:29,636 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:29,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,642 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:29,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,642 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:29,643 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:29,642 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832589706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832589707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832589707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832589707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832589711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,743 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:29,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:29,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:29,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:29,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:29,746 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:29,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:29,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:29,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,746 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,747 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:29,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:29,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:29,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:29,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,751 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa8b1f6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:29,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:29,752 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,753 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:29,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:29,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:29,754 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:29,756 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:29,758 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:29,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:29,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:29,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:29,758 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:29,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,761 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:29,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:29,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,764 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:29,764 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/17530a1c431f47a08211d25a521b0ce1 2023-04-07 01:55:29,864 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:29,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:29,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:29,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:29,868 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,869 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,869 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,870 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,870 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:29,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:29,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:29,871 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,871 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:29,872 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:29,874 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a574d35[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:29,876 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:29,876 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,877 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:29,879 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:29,881 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:29,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:29,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:29,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:29,881 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:29,883 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,883 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,883 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:29,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:29,885 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:29,888 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:29,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:29,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:29,888 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:29,888 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:29,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:29,989 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:29,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:29,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:29,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:29,996 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:29,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:29,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:29,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:29,999 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,000 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:30,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:30,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:30,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:30,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832590016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:30,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832590019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832590020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832590020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,031 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,031 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:30,031 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:30,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832590022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,035 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2d3e48[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:30,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:30,037 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:30,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:30,041 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:30,044 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:30,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:30,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:30,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:30,044 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:30,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:30,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:30,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:30,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:30,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:30,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:30,052 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:30,052 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:30,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,052 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:30,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,153 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:30,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:30,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:30,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:30,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:30,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:30,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:30,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:30,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:30,157 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,157 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:30,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:30,162 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,162 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:30,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:30,164 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,164 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:30,164 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:30,164 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:30,165 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,165 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:30,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454748f4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:30,167 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:30,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,169 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:30,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,169 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:30,172 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:30,175 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:30,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:30,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:30,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:30,175 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:30,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:30,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,179 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:30,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:30,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:30,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:30,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:30,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:30,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,182 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:30,182 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:30,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:30,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:30,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0539030de30248508f9ff2e3056c6e58 2023-04-07 01:55:30,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/56ac42ee9405433484423ce40c54bc24 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24 2023-04-07 01:55:30,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24, entries=200, sequenceid=883, filesize=32.3 K 2023-04-07 01:55:30,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/17530a1c431f47a08211d25a521b0ce1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1 2023-04-07 01:55:30,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1, entries=150, sequenceid=883, filesize=11.9 K 2023-04-07 01:55:30,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0539030de30248508f9ff2e3056c6e58 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58 2023-04-07 01:55:30,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58, entries=150, sequenceid=883, filesize=11.9 K 2023-04-07 01:55:30,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~327.56 KB/335424, currentSize=67.09 KB/68700 for 16a7d0fcb00e6e5b8493de38ebf26fec in 859ms, sequenceid=883, compaction requested=true 2023-04-07 01:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:30,219 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:30,219 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:30,220 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136208 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:30,220 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/A is initiating minor compaction (all files) 2023-04-07 01:55:30,220 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/A in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,220 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:30,220 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/ae5de6e639cf41b498f9919af7de8f05, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=133.0 K 2023-04-07 01:55:30,220 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/B is initiating minor compaction (all files) 2023-04-07 01:55:30,221 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/B in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,221 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e66f1fd2883247c3b50120794c27ed6a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=58.5 K 2023-04-07 01:55:30,221 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ae5de6e639cf41b498f9919af7de8f05, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523200 2023-04-07 01:55:30,221 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e66f1fd2883247c3b50120794c27ed6a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523200 2023-04-07 01:55:30,221 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6a32a757f96948fcb0e67123d5197cdd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680832524334 2023-04-07 01:55:30,221 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6c10a11e8b5f4b7da756c36f0acc0322, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680832524334 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting eef435bdba8c4c9d807becbeadd07f5d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1680832525480 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ecf5f724d8024c328fee27e23d2f90c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1680832525480 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3594862267ca434db2211fc2712ef6e1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832527640 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d17d0f7d5aac4eec935f4d2a08cec152, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832527640 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 56ac42ee9405433484423ce40c54bc24, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:30,222 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 17530a1c431f47a08211d25a521b0ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:30,231 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#B#compaction#366 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:30,232 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#A#compaction#365 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:30,282 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:30,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:30,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:30,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:30,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:30,285 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:30,285 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:30,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:30,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:30,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,287 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:30,287 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:30,288 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,288 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:30,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:30,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:30,289 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,289 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:30,289 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:55:30,289 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:30,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:30,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:30,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:30,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:30,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:30,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:30,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832590551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832590552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832590553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832590553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832590562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,647 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/39c9b37cc6fd4f0bafc777637140965a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/39c9b37cc6fd4f0bafc777637140965a 2023-04-07 01:55:30,651 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/B of 16a7d0fcb00e6e5b8493de38ebf26fec into 39c9b37cc6fd4f0bafc777637140965a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:30,651 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:30,651 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/B, priority=11, startTime=1680832530219; duration=0sec 2023-04-07 01:55:30,651 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:30,652 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:30,653 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:30,653 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/C is initiating minor compaction (all files) 2023-04-07 01:55:30,653 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/C in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:30,653 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/58764c13c3224da3a342aee5dc26f4ef, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=58.5 K 2023-04-07 01:55:30,653 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 58764c13c3224da3a342aee5dc26f4ef, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680832523200 2023-04-07 01:55:30,654 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ff27af94661241459b0ff14e3318a504, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680832524334 2023-04-07 01:55:30,654 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 74dd6c69b080447eb62af02102d1f86b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1680832525480 2023-04-07 01:55:30,654 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f9ae4bda74be40ffae4a283e225c2bad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680832527640 2023-04-07 01:55:30,655 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0539030de30248508f9ff2e3056c6e58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:30,662 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#C#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:30,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832590663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832590664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832590664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832590664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832590675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,704 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304070253b0c737ba458dad22268a7d131103 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070253b0c737ba458dad22268a7d131103 2023-04-07 01:55:30,713 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=896, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2bdf0cf17527485eb00e3118511643fa 2023-04-07 01:55:30,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832590874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832590874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832590874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832590874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:30,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832590886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,062 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/53338364464d449e84d19378451d438c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/53338364464d449e84d19378451d438c 2023-04-07 01:55:31,066 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/A of 16a7d0fcb00e6e5b8493de38ebf26fec into 53338364464d449e84d19378451d438c(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:31,066 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:31,066 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/A, priority=11, startTime=1680832530219; duration=0sec 2023-04-07 01:55:31,066 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:31,073 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/c770d9a4a3604dd2a5cffe4fcc5b1427 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/c770d9a4a3604dd2a5cffe4fcc5b1427 2023-04-07 01:55:31,077 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/C of 16a7d0fcb00e6e5b8493de38ebf26fec into c770d9a4a3604dd2a5cffe4fcc5b1427(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:31,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:31,077 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/C, priority=11, startTime=1680832530219; duration=0sec 2023-04-07 01:55:31,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:31,123 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/4f49713f83464eb3a60bc0451fb05039 2023-04-07 01:55:31,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832591181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832591181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832591183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832591183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832591193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,537 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0f4b979d36904ad38b7dfd4b3b571129 2023-04-07 01:55:31,543 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2bdf0cf17527485eb00e3118511643fa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa 2023-04-07 01:55:31,549 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa, entries=150, sequenceid=896, filesize=25.4 K 2023-04-07 01:55:31,550 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/4f49713f83464eb3a60bc0451fb05039 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039 2023-04-07 01:55:31,553 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039, entries=150, sequenceid=896, filesize=11.9 K 2023-04-07 01:55:31,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/0f4b979d36904ad38b7dfd4b3b571129 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129 2023-04-07 01:55:31,557 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129, entries=150, sequenceid=896, filesize=11.9 K 2023-04-07 01:55:31,558 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1269ms, sequenceid=896, compaction requested=false 2023-04-07 01:55:31,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:31,558 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:31,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2f66ab[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:31,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:31,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:31,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:31,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:31,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:31,567 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:31,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:31,572 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:31,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:31,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:31,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:31,572 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:31,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:31,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:31,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:31,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:31,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:31,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,579 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,582 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:31,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:31,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:31,582 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:31,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:31,583 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:31,583 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:31,683 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:31,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:31,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:31,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:31,686 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:31,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:31,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:31,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:31,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:31,688 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:31,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:31,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:31,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:31,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:31,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:31,690 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:31,690 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:31,690 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:31,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:31,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:31,690 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:31,690 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:31,690 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:31,691 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:31,691 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:31,691 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:31,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832591700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832591700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832591702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832591704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832591705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832591806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832591806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832591809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:31,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:31,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832591812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832592014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832592014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832592015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832592017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,107 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407bd8553c8161149dea102efe9972245ac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bd8553c8161149dea102efe9972245ac 2023-04-07 01:55:32,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832592319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832592319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832592319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832592322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,512 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=919, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/6228adb3ab504418ae988c5574cbfa9d 2023-04-07 01:55:32,524 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/92842e67b61248eb8b70ae796c2192c5 2023-04-07 01:55:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832592709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832592824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832592824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832592826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:32,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832592829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,933 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/5a8375b3c951411ab66e580783a9cf4b 2023-04-07 01:55:32,936 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/6228adb3ab504418ae988c5574cbfa9d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d 2023-04-07 01:55:32,940 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d, entries=200, sequenceid=919, filesize=32.3 K 2023-04-07 01:55:32,941 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/92842e67b61248eb8b70ae796c2192c5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5 2023-04-07 01:55:32,955 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5, entries=150, sequenceid=919, filesize=11.9 K 2023-04-07 01:55:32,956 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/5a8375b3c951411ab66e580783a9cf4b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b 2023-04-07 01:55:32,961 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b, entries=150, sequenceid=919, filesize=11.9 K 2023-04-07 01:55:32,962 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1272ms, sequenceid=919, compaction requested=true 2023-04-07 01:55:32,962 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:32,962 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:32,965 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a61bc3e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:32,968 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:32,968 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,969 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:32,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:32,979 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:32,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:32,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:32,982 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:32,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:32,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:32,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:32,982 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:32,984 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,984 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:32,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,984 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:32,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,989 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:32,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:32,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:32,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:32,992 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:32,992 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:32,992 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:32,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:32,993 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,993 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:32,993 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:32,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:33,093 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:33,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:33,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:33,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:33,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:33,096 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:33,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:33,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:33,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:33,097 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:33,097 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:33,098 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:33,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:33,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:33,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:33,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:33,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:33,100 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:33,100 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:33,516 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407055fa34c22ce46efaa8b74ffda52749b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407055fa34c22ce46efaa8b74ffda52749b 2023-04-07 01:55:33,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:33,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:33,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832593862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832593863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832593865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832593866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,922 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=931, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/8ca487bafbbe40289227c3c7bba6e628 2023-04-07 01:55:33,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832593968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832593974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832593974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:33,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:33,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832593974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832594175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832594182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832594183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832594183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,332 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=931 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/1ea3b573c3ae4a05877ccd5673a69b73 2023-04-07 01:55:34,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832594481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832594486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832594487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832594488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:34,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832594720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,724 DEBUG [Thread-2520] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:34,742 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=931 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/8ed806f968d14be796d4bec67830e454 2023-04-07 01:55:34,745 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/8ca487bafbbe40289227c3c7bba6e628 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628 2023-04-07 01:55:34,748 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628, entries=150, sequenceid=931, filesize=25.4 K 2023-04-07 01:55:34,749 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/1ea3b573c3ae4a05877ccd5673a69b73 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73 2023-04-07 01:55:34,752 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73, entries=150, sequenceid=931, filesize=11.9 K 2023-04-07 01:55:34,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/8ed806f968d14be796d4bec67830e454 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454 2023-04-07 01:55:34,755 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454, entries=150, sequenceid=931, filesize=11.9 K 2023-04-07 01:55:34,756 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1656ms, sequenceid=931, compaction requested=true 2023-04-07 01:55:34,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:34,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:34,760 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7b1244[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:34,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:34,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,763 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:34,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:34,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:34,763 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,763 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:34,765 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:34,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:34,767 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:34,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:34,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:34,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:34,767 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:34,770 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,770 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:34,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,770 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:34,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:34,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:34,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:34,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:34,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:34,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:34,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,774 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:34,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:34,774 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:34,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:34,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:34,874 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:34,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:34,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:34,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:34,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:34,877 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:34,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:34,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:34,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:34,878 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,878 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:34,879 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,879 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:34,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:34,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:34,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:34,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:34,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:34,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:34,881 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:34,881 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:34,882 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:34,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832595004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832595005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832595011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832595012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832595112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832595118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832595118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832595120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,296 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040763bb77565d324c959aa4a96fa8ef7768 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040763bb77565d324c959aa4a96fa8ef7768 2023-04-07 01:55:35,301 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=951, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/28f3cd314dbf4d698420bea64b4fd456 2023-04-07 01:55:35,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832595319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832595324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832595325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832595326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832595627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832595629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832595633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:35,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832595634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:35,714 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=951 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/e414e24b919a400586da32697d471325 2023-04-07 01:55:36,126 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=951 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/a46b7a6fcaa946cea517b8e474936a0b 2023-04-07 01:55:36,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/28f3cd314dbf4d698420bea64b4fd456 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456 2023-04-07 01:55:36,133 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456, entries=150, sequenceid=951, filesize=25.4 K 2023-04-07 01:55:36,134 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/e414e24b919a400586da32697d471325 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325 2023-04-07 01:55:36,137 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325, entries=150, sequenceid=951, filesize=11.9 K 2023-04-07 01:55:36,138 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/a46b7a6fcaa946cea517b8e474936a0b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b 2023-04-07 01:55:36,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832596134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,142 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b, entries=150, sequenceid=951, filesize=11.9 K 2023-04-07 01:55:36,142 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1261ms, sequenceid=951, compaction requested=true 2023-04-07 01:55:36,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:36,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:36,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:36,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:36,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:36,144 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:55:36,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:36,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:36,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4004edb7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:36,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:36,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:36,152 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:36,152 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:36,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,152 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:36,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:36,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:36,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:36,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:36,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:36,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:36,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:36,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:36,157 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,158 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,159 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:36,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:36,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,162 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:36,162 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:36,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:36,181 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040794b8f4184f974f6493387d4cea9d9585 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040794b8f4184f974f6493387d4cea9d9585 2023-04-07 01:55:36,205 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=964, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/237073fb5db4421d9eb76c643d4c678e 2023-04-07 01:55:36,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832596195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832596197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832596203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=964 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/a191e6000ada4803a6ff672d7baf60fb 2023-04-07 01:55:36,262 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:36,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:36,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:36,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:36,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:36,265 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:36,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:36,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:36,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,269 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,269 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,270 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,270 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:36,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:36,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:36,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:36,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:36,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:36,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:36,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:36,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:36,271 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:36,272 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,272 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316fd122[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:36,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:36,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:36,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:36,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:36,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:36,279 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:36,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:36,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:36,279 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:36,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:36,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,286 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,287 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,287 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:36,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:36,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,287 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:36,287 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:36,287 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,287 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,287 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832596307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832596308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832596309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,387 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:36,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:36,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:36,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:36,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:36,391 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:36,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:36,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:36,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:36,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:36,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:36,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:36,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,394 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:36,396 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:36,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46980b7d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:36,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,400 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:36,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:36,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:36,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:36,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,409 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,409 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:36,412 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:36,412 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:36,412 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:36,412 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:36,412 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:36,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,418 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:36,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,418 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:36,418 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:36,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,519 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:36,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:36,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:36,522 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:36,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:36,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:36,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,524 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832596519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832596520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:36,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:36,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:36,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:36,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:36,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:36,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:36,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2116c8d8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:36,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:36,531 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,532 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832596521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:36,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:36,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:36,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:36,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:36,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:36,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:36,537 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:36,537 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:36,537 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:36,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:36,538 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:36,539 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,539 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,539 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,540 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:36,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:36,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,544 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:36,544 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:36,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=964 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/bce6bfe9ea47491d94b9ae4b8f3342e1 2023-04-07 01:55:36,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/237073fb5db4421d9eb76c643d4c678e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e 2023-04-07 01:55:36,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e, entries=200, sequenceid=964, filesize=32.3 K 2023-04-07 01:55:36,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/a191e6000ada4803a6ff672d7baf60fb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb 2023-04-07 01:55:36,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb, entries=150, sequenceid=964, filesize=11.9 K 2023-04-07 01:55:36,644 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:36,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:36,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/bce6bfe9ea47491d94b9ae4b8f3342e1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1 2023-04-07 01:55:36,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:36,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1, entries=150, sequenceid=964, filesize=11.9 K 2023-04-07 01:55:36,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a7d0fcb00e6e5b8493de38ebf26fec in 510ms, sequenceid=964, compaction requested=true 2023-04-07 01:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:36,654 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:55:36,654 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:36,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:36,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170664 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:55:36,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/A is initiating minor compaction (all files) 2023-04-07 01:55:36,657 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/A in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,657 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/53338364464d449e84d19378451d438c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=166.7 K 2023-04-07 01:55:36,657 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73424 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:55:36,657 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 53338364464d449e84d19378451d438c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:36,657 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/B is initiating minor compaction (all files) 2023-04-07 01:55:36,657 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/B in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,657 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/39c9b37cc6fd4f0bafc777637140965a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=71.7 K 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2bdf0cf17527485eb00e3118511643fa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1680832529370 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 39c9b37cc6fd4f0bafc777637140965a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6228adb3ab504418ae988c5574cbfa9d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1680832530552 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4f49713f83464eb3a60bc0451fb05039, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1680832529370 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8ca487bafbbe40289227c3c7bba6e628, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=931, earliestPutTs=1680832531697 2023-04-07 01:55:36,658 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 92842e67b61248eb8b70ae796c2192c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1680832530552 2023-04-07 01:55:36,659 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 28f3cd314dbf4d698420bea64b4fd456, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=951, earliestPutTs=1680832533854 2023-04-07 01:55:36,659 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1ea3b573c3ae4a05877ccd5673a69b73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=931, earliestPutTs=1680832531697 2023-04-07 01:55:36,659 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 237073fb5db4421d9eb76c643d4c678e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535000 2023-04-07 01:55:36,659 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e414e24b919a400586da32697d471325, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=951, earliestPutTs=1680832533854 2023-04-07 01:55:36,659 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a191e6000ada4803a6ff672d7baf60fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535009 2023-04-07 01:55:36,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:36,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:36,665 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:36,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:36,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:36,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,667 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:36,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:36,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:36,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:36,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:36,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:36,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:36,669 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,669 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:36,669 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:36,670 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,683 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#B#compaction#384 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:36,723 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#A#compaction#383 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:36,742 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/04de9e2c6ec8467694b955a2bc028916 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/04de9e2c6ec8467694b955a2bc028916 2023-04-07 01:55:36,748 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/B of 16a7d0fcb00e6e5b8493de38ebf26fec into 04de9e2c6ec8467694b955a2bc028916(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:36,748 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:36,748 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/B, priority=10, startTime=1680832536654; duration=0sec 2023-04-07 01:55:36,748 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:36,749 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:55:36,752 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73424 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:55:36,752 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/C is initiating minor compaction (all files) 2023-04-07 01:55:36,752 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/C in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,752 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/c770d9a4a3604dd2a5cffe4fcc5b1427, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=71.7 K 2023-04-07 01:55:36,754 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c770d9a4a3604dd2a5cffe4fcc5b1427, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1680832529349 2023-04-07 01:55:36,754 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0f4b979d36904ad38b7dfd4b3b571129, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1680832529370 2023-04-07 01:55:36,756 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5a8375b3c951411ab66e580783a9cf4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1680832530552 2023-04-07 01:55:36,756 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8ed806f968d14be796d4bec67830e454, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=931, earliestPutTs=1680832531697 2023-04-07 01:55:36,758 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a46b7a6fcaa946cea517b8e474936a0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=951, earliestPutTs=1680832533854 2023-04-07 01:55:36,759 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bce6bfe9ea47491d94b9ae4b8f3342e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535009 2023-04-07 01:55:36,763 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a914eaad66324244b8d216387b128613 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a914eaad66324244b8d216387b128613 2023-04-07 01:55:36,799 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#C#compaction#386 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:36,800 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=983, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/d1fc974e6c1e4f25893fbb4c8bc5bf36 2023-04-07 01:55:36,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:36,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:36,839 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/abd72a37147a4c10a3c8c4f685b9c22e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/abd72a37147a4c10a3c8c4f685b9c22e 2023-04-07 01:55:36,842 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=983 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/051654291338465f885f0031a0af9c7b 2023-04-07 01:55:36,843 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/C of 16a7d0fcb00e6e5b8493de38ebf26fec into abd72a37147a4c10a3c8c4f685b9c22e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:36,843 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:36,844 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/C, priority=10, startTime=1680832536655; duration=0sec 2023-04-07 01:55:36,844 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:36,863 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=983 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/1c884271d4a6435dba5c026e258223f8 2023-04-07 01:55:36,868 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/d1fc974e6c1e4f25893fbb4c8bc5bf36 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36 2023-04-07 01:55:36,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832596863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,871 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36, entries=150, sequenceid=983, filesize=25.4 K 2023-04-07 01:55:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832596864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,872 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/051654291338465f885f0031a0af9c7b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b 2023-04-07 01:55:36,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:36,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832596865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,877 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b, entries=150, sequenceid=983, filesize=11.9 K 2023-04-07 01:55:36,879 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/1c884271d4a6435dba5c026e258223f8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8 2023-04-07 01:55:36,882 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8, entries=150, sequenceid=983, filesize=11.9 K 2023-04-07 01:55:36,883 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 16a7d0fcb00e6e5b8493de38ebf26fec in 214ms, sequenceid=983, compaction requested=false 2023-04-07 01:55:36,883 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:36,883 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:36,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:36,886 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:36,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4698e018[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:36,888 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:36,888 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,889 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:36,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:36,895 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:36,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:36,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:36,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:36,895 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:36,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,898 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:36,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,900 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:36,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:36,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:36,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:36,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:36,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:36,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:36,904 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:36,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:36,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:36,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:36,904 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:36,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:36,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:37,005 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:37,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:37,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:37,008 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:37,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:37,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:37,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,009 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:37,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:37,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:37,011 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,011 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:37,011 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:37,011 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:37,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:37,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b1cf28[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:37,016 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:37,016 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:37,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:37,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:37,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:37,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:37,023 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:37,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:37,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:37,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:37,024 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:37,027 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,027 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,028 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,030 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:37,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:37,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,034 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:37,034 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:37,034 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,035 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,035 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,135 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:37,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:37,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:37,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:37,138 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:37,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:37,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:37,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:37,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:37,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:37,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:37,142 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:37,142 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:37,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f74fe80[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:37,145 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:37,146 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:37,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:37,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:37,151 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:37,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:37,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:37,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:37,151 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:37,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832597148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,154 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,155 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:37,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,159 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,159 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:37,161 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:37,161 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,174 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/35c27924a549472ab7459a05e1a9b605 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/35c27924a549472ab7459a05e1a9b605 2023-04-07 01:55:37,178 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/A of 16a7d0fcb00e6e5b8493de38ebf26fec into 35c27924a549472ab7459a05e1a9b605(size=25.9 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:37,178 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:37,178 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/A, priority=10, startTime=1680832536654; duration=0sec 2023-04-07 01:55:37,178 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:37,261 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:37,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:37,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:37,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:37,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:37,275 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:37,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:37,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:37,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:37,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:37,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:37,278 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:37,279 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,279 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:37,282 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,283 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,283 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:37,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1833b7be[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:37,285 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:37,285 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:37,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:37,290 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:37,290 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:37,290 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:37,290 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:37,290 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:37,290 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:37,290 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:37,292 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,294 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:37,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,299 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,299 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,299 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:37,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,388 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304079a60ce0f87374ed5b2f6b5c41537f7ac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079a60ce0f87374ed5b2f6b5c41537f7ac 2023-04-07 01:55:37,399 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:37,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:37,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:37,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=999, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/c1337a729dff42e2a8961f85e739f3cf 2023-04-07 01:55:37,413 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:37,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:37,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:37,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:37,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:37,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:37,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:37,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:37,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:37,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:37,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:37,424 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,424 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:37,424 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:37,425 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:37,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192a1c80[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:37,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:37,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,430 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:37,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:37,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:37,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,432 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:37,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:37,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:37,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:37,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:37,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:37,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:37,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:37,439 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:37,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:37,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:37,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:37,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:37,446 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:37,447 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:37,447 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:37,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:37,463 WARN [DataStreamer for file /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ee0214ddbce04303b5bccd7a2d12b5fb] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-07 01:55:37,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=999 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ee0214ddbce04303b5bccd7a2d12b5fb 2023-04-07 01:55:37,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=999 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/da3ea69797b649cf9cc3402edf61bfba 2023-04-07 01:55:37,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/c1337a729dff42e2a8961f85e739f3cf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf 2023-04-07 01:55:37,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf, entries=200, sequenceid=999, filesize=32.3 K 2023-04-07 01:55:37,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/ee0214ddbce04303b5bccd7a2d12b5fb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb 2023-04-07 01:55:37,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb, entries=150, sequenceid=999, filesize=11.9 K 2023-04-07 01:55:37,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/da3ea69797b649cf9cc3402edf61bfba as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba 2023-04-07 01:55:37,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba, entries=150, sequenceid=999, filesize=11.9 K 2023-04-07 01:55:37,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 16a7d0fcb00e6e5b8493de38ebf26fec in 544ms, sequenceid=999, compaction requested=true 2023-04-07 01:55:37,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:37,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:37,517 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:37,517 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:37,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:37,518 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:37,518 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:37,519 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/A is initiating minor compaction (all files) 2023-04-07 01:55:37,519 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/A in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,519 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/35c27924a549472ab7459a05e1a9b605, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=83.7 K 2023-04-07 01:55:37,519 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:37,519 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/B is initiating minor compaction (all files) 2023-04-07 01:55:37,519 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/B in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,519 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/04de9e2c6ec8467694b955a2bc028916, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=36.2 K 2023-04-07 01:55:37,519 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 35c27924a549472ab7459a05e1a9b605, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535009 2023-04-07 01:55:37,520 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 04de9e2c6ec8467694b955a2bc028916, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535009 2023-04-07 01:55:37,520 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d1fc974e6c1e4f25893fbb4c8bc5bf36, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=983, earliestPutTs=1680832536193 2023-04-07 01:55:37,520 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 051654291338465f885f0031a0af9c7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=983, earliestPutTs=1680832536193 2023-04-07 01:55:37,520 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c1337a729dff42e2a8961f85e739f3cf, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:37,520 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ee0214ddbce04303b5bccd7a2d12b5fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:37,539 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#B#compaction#392 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:37,547 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:37,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:37,576 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:37,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:37,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:37,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:37,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:37,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:37,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:37,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:37,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:37,581 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,581 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:37,581 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:55:37,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:37,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:37,583 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:37,583 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:37,583 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:37,583 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:37,589 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#A#compaction#393 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:37,604 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/8638bcde0bad42e3b79086fcb37f07dc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8638bcde0bad42e3b79086fcb37f07dc 2023-04-07 01:55:37,610 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/B of 16a7d0fcb00e6e5b8493de38ebf26fec into 8638bcde0bad42e3b79086fcb37f07dc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:37,611 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:37,611 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/B, priority=13, startTime=1680832537517; duration=0sec 2023-04-07 01:55:37,611 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:37,611 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:37,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:37,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/C is initiating minor compaction (all files) 2023-04-07 01:55:37,612 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/C in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:37,612 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/abd72a37147a4c10a3c8c4f685b9c22e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=36.2 K 2023-04-07 01:55:37,612 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting abd72a37147a4c10a3c8c4f685b9c22e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=964, earliestPutTs=1680832535009 2023-04-07 01:55:37,613 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1c884271d4a6435dba5c026e258223f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=983, earliestPutTs=1680832536193 2023-04-07 01:55:37,614 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting da3ea69797b649cf9cc3402edf61bfba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:37,621 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#C#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:37,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:37,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:37,645 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040733adb6fbf9fe47c18e6d73e9ab1f1d82 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040733adb6fbf9fe47c18e6d73e9ab1f1d82 2023-04-07 01:55:37,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832597974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832597974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:37,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:37,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832597979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,059 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/8ac2aa9e6b6b412d86e722fb48b1c5ca as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ac2aa9e6b6b412d86e722fb48b1c5ca 2023-04-07 01:55:38,060 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1018, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2f9bd17011df4cd788533bbd540e3095 2023-04-07 01:55:38,063 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/C of 16a7d0fcb00e6e5b8493de38ebf26fec into 8ac2aa9e6b6b412d86e722fb48b1c5ca(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:38,063 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:38,063 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/C, priority=13, startTime=1680832537518; duration=0sec 2023-04-07 01:55:38,064 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:38,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832598278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832598279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832598284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,435 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/77eb2262f3db4e5fb09dc965cafe4819 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/77eb2262f3db4e5fb09dc965cafe4819 2023-04-07 01:55:38,438 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/A of 16a7d0fcb00e6e5b8493de38ebf26fec into 77eb2262f3db4e5fb09dc965cafe4819(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:38,439 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:38,439 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/A, priority=13, startTime=1680832537517; duration=0sec 2023-04-07 01:55:38,439 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:38,470 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/fccf1c0c06ca44f7a7901809af8bf7e8 2023-04-07 01:55:38,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38434 deadline: 1680832598740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,744 DEBUG [Thread-2520] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:55:38,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832598786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832598788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:38,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832598791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,880 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/68798427bd6749a090bd352d088fd219 2023-04-07 01:55:38,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2f9bd17011df4cd788533bbd540e3095 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095 2023-04-07 01:55:38,888 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095, entries=150, sequenceid=1018, filesize=25.4 K 2023-04-07 01:55:38,889 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/fccf1c0c06ca44f7a7901809af8bf7e8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8 2023-04-07 01:55:38,893 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8, entries=150, sequenceid=1018, filesize=11.9 K 2023-04-07 01:55:38,894 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/68798427bd6749a090bd352d088fd219 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219 2023-04-07 01:55:38,897 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219, entries=150, sequenceid=1018, filesize=11.9 K 2023-04-07 01:55:38,898 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1317ms, sequenceid=1018, compaction requested=false 2023-04-07 01:55:38,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:38,898 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:38,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:38,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:38,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:38,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ee55802[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,903 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:38,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:38,903 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:38,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:38,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:38,908 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:38,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:38,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:38,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:38,908 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:38,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:38,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:38,910 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:38,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:38,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:38,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:38,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:38,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:38,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:38,914 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:38,913 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:38,914 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:38,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:38,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:38,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:38,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:39,014 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:39,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:39,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:39,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:39,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:39,017 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:39,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:39,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:39,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:39,019 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,019 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:39,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:39,021 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,021 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:39,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:39,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:39,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:39,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:39,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:39,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:39,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:39,022 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:39,022 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:39,023 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:39,023 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,040 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407459ae852d681433fb5721f1cfb69137c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407459ae852d681433fb5721f1cfb69137c 2023-04-07 01:55:39,047 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1035, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/15b9e3ad2f4646919eab2b12fb71f039 2023-04-07 01:55:39,057 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1035 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/775a4f45cd4b45b494615909ed16e08d 2023-04-07 01:55:39,067 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1035 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/15579a2b1ea14b089435f93d62240a62 2023-04-07 01:55:39,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/15b9e3ad2f4646919eab2b12fb71f039 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039 2023-04-07 01:55:39,077 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039, entries=150, sequenceid=1035, filesize=25.4 K 2023-04-07 01:55:39,078 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/775a4f45cd4b45b494615909ed16e08d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d 2023-04-07 01:55:39,081 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d, entries=150, sequenceid=1035, filesize=11.9 K 2023-04-07 01:55:39,082 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/15579a2b1ea14b089435f93d62240a62 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62 2023-04-07 01:55:39,086 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62, entries=150, sequenceid=1035, filesize=11.9 K 2023-04-07 01:55:39,087 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 16a7d0fcb00e6e5b8493de38ebf26fec in 64ms, sequenceid=1035, compaction requested=true 2023-04-07 01:55:39,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:39,087 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:39,088 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:39,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:39,096 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:39,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:39,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@364d25a0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:39,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:39,098 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,099 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:39,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:39,099 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,099 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:39,099 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:39,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:39,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:39,101 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:39,104 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:39,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:39,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:39,104 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:39,104 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:39,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:39,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,107 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:39,107 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:39,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:39,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:39,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:39,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:39,111 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:39,111 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:39,111 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:39,111 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:39,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:39,212 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:39,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:39,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:39,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:39,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:39,215 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:39,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:39,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:39,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:39,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:39,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:39,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:39,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:39,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:39,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:39,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:39,218 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:39,218 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:39,219 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:39,219 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:39,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:39,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:39,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832539369, max timestamp=1680832539386Num uniques 859; 2023-04-07 01:55:39,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:39,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832539369, max timestamp=1680832539386Num uniques 859; 2023-04-07 01:55:39,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832539369, max timestamp=1680832539386Num uniques 859; 2023-04-07 01:55:39,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:39,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832539453, max timestamp=1680832539466Num uniques 1018; 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832539453, max timestamp=1680832539466Num uniques 1018; 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832539453, max timestamp=1680832539466Num uniques 1018; 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:39,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:55:39,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832539527, max timestamp=1680832539558Num uniques 1027; 2023-04-07 01:55:39,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:39,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832539527, max timestamp=1680832539558Num uniques 1027; 2023-04-07 01:55:39,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:39,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832539527, max timestamp=1680832539558Num uniques 1027; 2023-04-07 01:55:39,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:55:39,630 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304072b8a0d1df6274324a233ba95674825f3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072b8a0d1df6274324a233ba95674825f3 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832539625, max timestamp=1680832539667Num uniques 1036; 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832539625, max timestamp=1680832539667Num uniques 1036; 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680832539625, max timestamp=1680832539667Num uniques 1036; 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:39,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-07 01:55:39,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832539738, max timestamp=1680832539751Num uniques 1045; 2023-04-07 01:55:39,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:39,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832539738, max timestamp=1680832539751Num uniques 1045; 2023-04-07 01:55:39,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:39,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832539738, max timestamp=1680832539751Num uniques 1045; 2023-04-07 01:55:39,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832539804, max timestamp=1680832539827Num uniques 1054; 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832539804, max timestamp=1680832539827Num uniques 1054; 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832539804, max timestamp=1680832539827Num uniques 1054; 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:39,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1680832539849, max timestamp=1680832539861Num uniques 1013; 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832539849, max timestamp=1680832539861Num uniques 1013; 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1680832539849, max timestamp=1680832539861Num uniques 1013; 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:39,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832539882, max timestamp=1680832539890Num uniques 1022; 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832539882, max timestamp=1680832539890Num uniques 1022; 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680832539882, max timestamp=1680832539890Num uniques 1022; 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 01:55:39,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 01:55:39,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832539904, max timestamp=1680832539923Num uniques 1081; 2023-04-07 01:55:39,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-07 01:55:39,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832539904, max timestamp=1680832539923Num uniques 1081; 2023-04-07 01:55:39,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680832539904, max timestamp=1680832539923Num uniques 1081; 2023-04-07 01:55:39,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-07 01:55:39,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13270, totalHeapSize=30400, min timestamp=1680832539939, max timestamp=1680832539992Num uniques 1090; 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=13270, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13272, totalHeapSize=30400, min timestamp=1680832539939, max timestamp=1680832539992Num uniques 1090; 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13276, totalHeapSize=30400, min timestamp=1680832539939, max timestamp=1680832539992Num uniques 1090; 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49916, new segment data size=13276, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-07 01:55:39,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49912, new segment data size=13272, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-07 01:55:40,035 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1039, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/b1e5ab085cd74352890ebedb9eb1c212 2023-04-07 01:55:40,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680832540019, max timestamp=1680832540033Num uniques 1149; 2023-04-07 01:55:40,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-07 01:55:40,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680832540019, max timestamp=1680832540033Num uniques 1149; 2023-04-07 01:55:40,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-07 01:55:40,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11399, totalHeapSize=26136, min timestamp=1680832540019, max timestamp=1680832540034Num uniques 1149; 2023-04-07 01:55:40,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52619, new segment data size=11399, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-07 01:55:40,045 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1039 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/5a1c9cd5a6324fbea26f14319b6f5064 2023-04-07 01:55:40,054 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1039 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/aabf202d4b8f49c3a5229aecd292cfc0 2023-04-07 01:55:40,058 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/b1e5ab085cd74352890ebedb9eb1c212 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212 2023-04-07 01:55:40,062 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212, entries=50, sequenceid=1039, filesize=11.7 K 2023-04-07 01:55:40,063 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/5a1c9cd5a6324fbea26f14319b6f5064 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064 2023-04-07 01:55:40,066 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064, entries=50, sequenceid=1039, filesize=7.1 K 2023-04-07 01:55:40,067 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/aabf202d4b8f49c3a5229aecd292cfc0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0 2023-04-07 01:55:40,070 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0, entries=50, sequenceid=1039, filesize=7.1 K 2023-04-07 01:55:40,070 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=134.18 KB/137400 for 16a7d0fcb00e6e5b8493de38ebf26fec in 851ms, sequenceid=1039, compaction requested=true 2023-04-07 01:55:40,070 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:40,070 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,072 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c6ea90[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,075 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:40,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=147.60 KB heapSize=416.39 KB 2023-04-07 01:55:40,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,076 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:55:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:55:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:40,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 01:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,081 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,081 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,092 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,092 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,093 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,093 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,093 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832600088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832600088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832600089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832600089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,193 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,196 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,198 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,198 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,200 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,200 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,200 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,200 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,203 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,203 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,203 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,203 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a413e74[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,206 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832600202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832600202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832600202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832600202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,210 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,211 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,212 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,212 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,212 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,213 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,214 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:40,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,217 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,218 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,218 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,318 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,321 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,322 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e505c76[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,329 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,332 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,335 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,335 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,335 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,336 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,336 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,336 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,338 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,340 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,340 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,340 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832600410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832600410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832600410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832600411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,440 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,447 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,449 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,451 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,452 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,452 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,452 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,452 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,456 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116f1039[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,456 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,457 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,459 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,463 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,464 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,466 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,466 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,466 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,471 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,471 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,472 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,472 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,472 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,490 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f5bd268bbc4048a1bbd5228a3da79943 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f5bd268bbc4048a1bbd5228a3da79943 2023-04-07 01:55:40,572 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,576 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,579 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f9eaee[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,586 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,592 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,592 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,592 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,595 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,597 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,598 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,598 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,598 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,598 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,598 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,698 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,701 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,704 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,704 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,704 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,708 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6638c4bb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,711 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,713 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832600713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,722 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:40,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,723 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,723 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832600716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832600717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:40,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832600724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,823 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,826 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,828 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,829 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,829 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,833 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,833 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4797e6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,834 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,836 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,837 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,837 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,838 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,840 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,840 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,841 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,841 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,841 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,845 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,845 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,845 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,845 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,845 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,846 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,846 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,903 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1250, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2e25c70fe0b94dcda9122b553aacad05 2023-04-07 01:55:40,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1250 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/1000bc9bb5fb4f979c7d52946eb17271 2023-04-07 01:55:40,946 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:40,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:40,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:40,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:40,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:40,950 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:40,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:40,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:40,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,950 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,952 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:40,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:40,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:40,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:40,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:40,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:40,954 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:40,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,954 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:40,955 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:40,955 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:40,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:40,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:40,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:40,955 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:40,956 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,956 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:40,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c0b44f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:40,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:40,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:40,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:40,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:40,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:40,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:40,970 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:40,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:40,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:40,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:40,970 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:40,972 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,972 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:40,972 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,972 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:40,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:40,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:40,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:40,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:40,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:40,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:40,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:40,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:40,977 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:40,977 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:40,977 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:40,977 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:40,977 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,077 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:41,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:41,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:41,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:41,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:41,080 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:41,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:41,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:41,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,082 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,082 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,082 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,082 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:41,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:41,083 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:41,083 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,083 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:41,084 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,084 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:41,084 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:41,084 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:41,086 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,086 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b49843[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:41,088 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:41,088 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:41,089 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,089 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:41,091 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:41,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,091 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:41,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:41,093 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:41,093 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:41,093 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:41,093 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:41,093 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:41,095 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,095 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,095 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,095 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,096 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,100 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:41,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:41,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:41,100 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:41,100 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,100 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,101 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,200 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:41,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:41,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:41,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:41,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:41,203 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:41,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:41,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:41,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,205 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:41,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:41,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:41,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:41,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:41,207 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:41,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:41,209 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39099ad2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:41,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:41,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:41,212 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,212 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:41,217 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:41,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:41,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:41,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:41,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:41,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:41,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,221 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,222 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:41,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:41,225 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:41,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,225 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:41,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:41,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38408 deadline: 1680832601220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:41,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38456 deadline: 1680832601226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:41,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38396 deadline: 1680832601228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:41,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38468 deadline: 1680832601229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1250 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/523e0f8a0f88487698c120dcc3621f26 2023-04-07 01:55:41,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/2e25c70fe0b94dcda9122b553aacad05 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05 2023-04-07 01:55:41,325 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:41,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:41,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:41,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:41,329 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:41,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:41,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:41,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05, entries=150, sequenceid=1250, filesize=25.4 K 2023-04-07 01:55:41,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/1000bc9bb5fb4f979c7d52946eb17271 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271 2023-04-07 01:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:41,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:41,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:41,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:41,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:41,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. started... 2023-04-07 01:55:41,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. as already flushing 2023-04-07 01:55:41,333 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:41,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:41,335 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271, entries=150, sequenceid=1250, filesize=11.9 K 2023-04-07 01:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/523e0f8a0f88487698c120dcc3621f26 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26 2023-04-07 01:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:41,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f67d2b2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:41,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:41,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:41,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:41,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:41,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:41,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26, entries=150, sequenceid=1250, filesize=11.9 K 2023-04-07 01:55:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,340 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~456.89 KB/467856, currentSize=26.84 KB/27480 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1265ms, sequenceid=1250, compaction requested=true 2023-04-07 01:55:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:41,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,341 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:41,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:41,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:41,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 01:55:41,341 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56658 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:41,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/B is initiating minor compaction (all files) 2023-04-07 01:55:41,343 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/B in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,343 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8638bcde0bad42e3b79086fcb37f07dc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=55.3 K 2023-04-07 01:55:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 116754 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:41,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/A is initiating minor compaction (all files) 2023-04-07 01:55:41,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8638bcde0bad42e3b79086fcb37f07dc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:41,343 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/A in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,343 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/77eb2262f3db4e5fb09dc965cafe4819, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=114.0 K 2023-04-07 01:55:41,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,344 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 77eb2262f3db4e5fb09dc965cafe4819, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:41,344 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fccf1c0c06ca44f7a7901809af8bf7e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1018, earliestPutTs=1680832536998 2023-04-07 01:55:41,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:41,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:41,344 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2f9bd17011df4cd788533bbd540e3095, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1018, earliestPutTs=1680832536998 2023-04-07 01:55:41,344 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:41,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:41,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:41,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:41,344 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:41,344 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 775a4f45cd4b45b494615909ed16e08d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680832537654 2023-04-07 01:55:41,345 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 15b9e3ad2f4646919eab2b12fb71f039, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680832537654 2023-04-07 01:55:41,345 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b1e5ab085cd74352890ebedb9eb1c212, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=1039, earliestPutTs=1680832539201 2023-04-07 01:55:41,345 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5a1c9cd5a6324fbea26f14319b6f5064, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=1039, earliestPutTs=1680832539201 2023-04-07 01:55:41,346 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2e25c70fe0b94dcda9122b553aacad05, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1250, earliestPutTs=1680832540064 2023-04-07 01:55:41,346 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1000bc9bb5fb4f979c7d52946eb17271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1250, earliestPutTs=1680832540064 2023-04-07 01:55:41,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,349 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:41,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,350 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:41,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:41,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:41,356 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#B#compaction#408 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:41,357 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:41,357 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:41,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:41,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:41,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:41,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:41,357 DEBUG [Thread-2528] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:41,357 DEBUG [Thread-2528] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:41,358 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#A#compaction#407 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:41,381 DEBUG [Thread-2535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:61971 2023-04-07 01:55:41,381 DEBUG [Thread-2529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:61971 2023-04-07 01:55:41,381 DEBUG [Thread-2535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:41,381 DEBUG [Thread-2529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:41,381 DEBUG [Thread-2531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:61971 2023-04-07 01:55:41,381 DEBUG [Thread-2531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:41,415 DEBUG [Thread-2537] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:61971 2023-04-07 01:55:41,415 DEBUG [Thread-2533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:61971 2023-04-07 01:55:41,415 DEBUG [Thread-2537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:41,415 DEBUG [Thread-2533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:41,458 DEBUG [Thread-2528] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:41,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:41,766 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/8f2fe24ab6f244ec845f587a1b01e972 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8f2fe24ab6f244ec845f587a1b01e972 2023-04-07 01:55:41,769 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/B of 16a7d0fcb00e6e5b8493de38ebf26fec into 8f2fe24ab6f244ec845f587a1b01e972(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:41,769 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:41,770 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/B, priority=11, startTime=1680832541341; duration=0sec 2023-04-07 01:55:41,770 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:41,770 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:55:41,771 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56658 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:55:41,771 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 16a7d0fcb00e6e5b8493de38ebf26fec/C is initiating minor compaction (all files) 2023-04-07 01:55:41,771 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a7d0fcb00e6e5b8493de38ebf26fec/C in TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:41,771 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ac2aa9e6b6b412d86e722fb48b1c5ca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp, totalSize=55.3 K 2023-04-07 01:55:41,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8ac2aa9e6b6b412d86e722fb48b1c5ca, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=999, earliestPutTs=1680832536846 2023-04-07 01:55:41,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 68798427bd6749a090bd352d088fd219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1018, earliestPutTs=1680832536998 2023-04-07 01:55:41,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 15579a2b1ea14b089435f93d62240a62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1035, earliestPutTs=1680832537654 2023-04-07 01:55:41,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting aabf202d4b8f49c3a5229aecd292cfc0, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=1039, earliestPutTs=1680832539201 2023-04-07 01:55:41,773 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 523e0f8a0f88487698c120dcc3621f26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1250, earliestPutTs=1680832540064 2023-04-07 01:55:41,779 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a7d0fcb00e6e5b8493de38ebf26fec#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:42,175 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/16a7a47ef0014d478c6ba55e52258383 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/16a7a47ef0014d478c6ba55e52258383 2023-04-07 01:55:42,178 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/A of 16a7d0fcb00e6e5b8493de38ebf26fec into 16a7a47ef0014d478c6ba55e52258383(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:42,178 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:42,178 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/A, priority=11, startTime=1680832541341; duration=0sec 2023-04-07 01:55:42,178 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:42,185 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/83fdd194fed94ed3bf21dceb1f2a09b6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/83fdd194fed94ed3bf21dceb1f2a09b6 2023-04-07 01:55:42,188 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a7d0fcb00e6e5b8493de38ebf26fec/C of 16a7d0fcb00e6e5b8493de38ebf26fec into 83fdd194fed94ed3bf21dceb1f2a09b6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:42,188 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:42,188 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec., storeName=16a7d0fcb00e6e5b8493de38ebf26fec/C, priority=11, startTime=1680832541341; duration=0sec 2023-04-07 01:55:42,188 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:42,230 DEBUG [Thread-2522] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:61971 2023-04-07 01:55:42,230 DEBUG [Thread-2522] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:42,234 DEBUG [Thread-2524] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:61971 2023-04-07 01:55:42,234 DEBUG [Thread-2524] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:42,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:42,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:55:42,240 DEBUG [Thread-2518] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:61971 2023-04-07 01:55:42,240 DEBUG [Thread-2518] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:42,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:42,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:42,240 DEBUG [Thread-2526] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:61971 2023-04-07 01:55:42,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:42,240 DEBUG [Thread-2526] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:42,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:42,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:42,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:42,650 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040778854a4986e14b30b60a45af8df90cdc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040778854a4986e14b30b60a45af8df90cdc 2023-04-07 01:55:43,054 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1264, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/518aad10485a4931b5c63eea4dc65839 2023-04-07 01:55:43,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1264 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/08411017d27b43bbb588d03fa949bd94 2023-04-07 01:55:43,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1264 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/3a37445f325748cfa8349f97edd805bb 2023-04-07 01:55:43,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/518aad10485a4931b5c63eea4dc65839 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/518aad10485a4931b5c63eea4dc65839 2023-04-07 01:55:43,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/518aad10485a4931b5c63eea4dc65839, entries=100, sequenceid=1264, filesize=18.6 K 2023-04-07 01:55:43,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/08411017d27b43bbb588d03fa949bd94 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/08411017d27b43bbb588d03fa949bd94 2023-04-07 01:55:43,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/08411017d27b43bbb588d03fa949bd94, entries=100, sequenceid=1264, filesize=9.5 K 2023-04-07 01:55:43,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/3a37445f325748cfa8349f97edd805bb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/3a37445f325748cfa8349f97edd805bb 2023-04-07 01:55:43,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/3a37445f325748cfa8349f97edd805bb, entries=100, sequenceid=1264, filesize=9.5 K 2023-04-07 01:55:43,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1647ms, sequenceid=1264, compaction requested=false 2023-04-07 01:55:43,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:44,317 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-07 01:55:44,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/ae5de6e639cf41b498f9919af7de8f05, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/53338364464d449e84d19378451d438c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/35c27924a549472ab7459a05e1a9b605, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/77eb2262f3db4e5fb09dc965cafe4819, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05] to archive 2023-04-07 01:55:44,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:44,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/e60c2cba74d94ef4843a6af911d66100 2023-04-07 01:55:44,339 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/fe00839e55d446ae91f2e1c8f96d7dc2 2023-04-07 01:55:44,340 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/9663c672bd1943b483881b30fbc8a704 2023-04-07 01:55:44,341 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/331a9d79358a43c2878f1502e8b6b51d 2023-04-07 01:55:44,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/ae5de6e639cf41b498f9919af7de8f05 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/ae5de6e639cf41b498f9919af7de8f05 2023-04-07 01:55:44,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6a32a757f96948fcb0e67123d5197cdd 2023-04-07 01:55:44,343 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/eef435bdba8c4c9d807becbeadd07f5d 2023-04-07 01:55:44,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/3594862267ca434db2211fc2712ef6e1 2023-04-07 01:55:44,345 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/56ac42ee9405433484423ce40c54bc24 2023-04-07 01:55:44,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/53338364464d449e84d19378451d438c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/53338364464d449e84d19378451d438c 2023-04-07 01:55:44,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2bdf0cf17527485eb00e3118511643fa 2023-04-07 01:55:44,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/6228adb3ab504418ae988c5574cbfa9d 2023-04-07 01:55:44,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/8ca487bafbbe40289227c3c7bba6e628 2023-04-07 01:55:44,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/28f3cd314dbf4d698420bea64b4fd456 2023-04-07 01:55:44,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/237073fb5db4421d9eb76c643d4c678e 2023-04-07 01:55:44,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/35c27924a549472ab7459a05e1a9b605 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/35c27924a549472ab7459a05e1a9b605 2023-04-07 01:55:44,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/d1fc974e6c1e4f25893fbb4c8bc5bf36 2023-04-07 01:55:44,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/c1337a729dff42e2a8961f85e739f3cf 2023-04-07 01:55:44,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/77eb2262f3db4e5fb09dc965cafe4819 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/77eb2262f3db4e5fb09dc965cafe4819 2023-04-07 01:55:44,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2f9bd17011df4cd788533bbd540e3095 2023-04-07 01:55:44,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/15b9e3ad2f4646919eab2b12fb71f039 2023-04-07 01:55:44,355 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/b1e5ab085cd74352890ebedb9eb1c212 2023-04-07 01:55:44,355 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/2e25c70fe0b94dcda9122b553aacad05 2023-04-07 01:55:44,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e66f1fd2883247c3b50120794c27ed6a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/39c9b37cc6fd4f0bafc777637140965a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/04de9e2c6ec8467694b955a2bc028916, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8638bcde0bad42e3b79086fcb37f07dc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271] to archive 2023-04-07 01:55:44,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:44,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/56280fdd1f2b46faaa6b242c54be04d1 2023-04-07 01:55:44,380 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ae46b7620bda403fb873abcf63c4f7e0 2023-04-07 01:55:44,381 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b37c6ed2d03748bdb350deb74abf5647 2023-04-07 01:55:44,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e66f1fd2883247c3b50120794c27ed6a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e66f1fd2883247c3b50120794c27ed6a 2023-04-07 01:55:44,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/dc45c2d86e8f4ea5802e11dcc6c2101f 2023-04-07 01:55:44,384 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/6c10a11e8b5f4b7da756c36f0acc0322 2023-04-07 01:55:44,385 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ecf5f724d8024c328fee27e23d2f90c9 2023-04-07 01:55:44,386 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/d17d0f7d5aac4eec935f4d2a08cec152 2023-04-07 01:55:44,387 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/39c9b37cc6fd4f0bafc777637140965a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/39c9b37cc6fd4f0bafc777637140965a 2023-04-07 01:55:44,388 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/17530a1c431f47a08211d25a521b0ce1 2023-04-07 01:55:44,389 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/4f49713f83464eb3a60bc0451fb05039 2023-04-07 01:55:44,389 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/92842e67b61248eb8b70ae796c2192c5 2023-04-07 01:55:44,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1ea3b573c3ae4a05877ccd5673a69b73 2023-04-07 01:55:44,391 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/e414e24b919a400586da32697d471325 2023-04-07 01:55:44,392 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/04de9e2c6ec8467694b955a2bc028916 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/04de9e2c6ec8467694b955a2bc028916 2023-04-07 01:55:44,393 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/a191e6000ada4803a6ff672d7baf60fb 2023-04-07 01:55:44,394 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/051654291338465f885f0031a0af9c7b 2023-04-07 01:55:44,394 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8638bcde0bad42e3b79086fcb37f07dc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8638bcde0bad42e3b79086fcb37f07dc 2023-04-07 01:55:44,395 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/ee0214ddbce04303b5bccd7a2d12b5fb 2023-04-07 01:55:44,396 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/fccf1c0c06ca44f7a7901809af8bf7e8 2023-04-07 01:55:44,397 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/775a4f45cd4b45b494615909ed16e08d 2023-04-07 01:55:44,398 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/5a1c9cd5a6324fbea26f14319b6f5064 2023-04-07 01:55:44,398 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/1000bc9bb5fb4f979c7d52946eb17271 2023-04-07 01:55:44,402 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/58764c13c3224da3a342aee5dc26f4ef, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/c770d9a4a3604dd2a5cffe4fcc5b1427, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/abd72a37147a4c10a3c8c4f685b9c22e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ac2aa9e6b6b412d86e722fb48b1c5ca, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26] to archive 2023-04-07 01:55:44,403 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:55:44,404 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f227e39896dd47cdb82937b200aa2b29 2023-04-07 01:55:44,405 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/593525f874234ec5bba42db604db5b99 2023-04-07 01:55:44,406 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/e83232c3427c4afca4e96bbb1e3815d3 2023-04-07 01:55:44,407 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/58764c13c3224da3a342aee5dc26f4ef to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/58764c13c3224da3a342aee5dc26f4ef 2023-04-07 01:55:44,407 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0893bd2e19b3471db4b5c640e849cd5c 2023-04-07 01:55:44,408 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/ff27af94661241459b0ff14e3318a504 2023-04-07 01:55:44,409 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/74dd6c69b080447eb62af02102d1f86b 2023-04-07 01:55:44,410 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/f9ae4bda74be40ffae4a283e225c2bad 2023-04-07 01:55:44,410 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/c770d9a4a3604dd2a5cffe4fcc5b1427 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/c770d9a4a3604dd2a5cffe4fcc5b1427 2023-04-07 01:55:44,411 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0539030de30248508f9ff2e3056c6e58 2023-04-07 01:55:44,412 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/0f4b979d36904ad38b7dfd4b3b571129 2023-04-07 01:55:44,413 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5a8375b3c951411ab66e580783a9cf4b 2023-04-07 01:55:44,414 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ed806f968d14be796d4bec67830e454 2023-04-07 01:55:44,415 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/a46b7a6fcaa946cea517b8e474936a0b 2023-04-07 01:55:44,416 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/abd72a37147a4c10a3c8c4f685b9c22e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/abd72a37147a4c10a3c8c4f685b9c22e 2023-04-07 01:55:44,417 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/bce6bfe9ea47491d94b9ae4b8f3342e1 2023-04-07 01:55:44,418 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/1c884271d4a6435dba5c026e258223f8 2023-04-07 01:55:44,419 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ac2aa9e6b6b412d86e722fb48b1c5ca to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/8ac2aa9e6b6b412d86e722fb48b1c5ca 2023-04-07 01:55:44,420 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/da3ea69797b649cf9cc3402edf61bfba 2023-04-07 01:55:44,421 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/68798427bd6749a090bd352d088fd219 2023-04-07 01:55:44,422 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/15579a2b1ea14b089435f93d62240a62 2023-04-07 01:55:44,423 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/aabf202d4b8f49c3a5229aecd292cfc0 2023-04-07 01:55:44,423 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/523e0f8a0f88487698c120dcc3621f26 2023-04-07 01:55:48,844 DEBUG [Thread-2520] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:61971 2023-04-07 01:55:48,844 DEBUG [Thread-2520] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 307 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 162 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 228 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 245 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 241 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1818 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5454 rows 2023-04-07 01:55:48,844 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1831 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5491 rows 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1845 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5532 rows 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1832 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5496 rows 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 1826 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 5475 rows 2023-04-07 01:55:48,845 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:55:48,845 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:61971 2023-04-07 01:55:48,845 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:55:48,846 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:55:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:55:48,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:48,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:48,853 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832548853"}]},"ts":"1680832548853"} 2023-04-07 01:55:48,854 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:55:48,855 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:55:48,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, UNASSIGN}] 2023-04-07 01:55:48,857 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, UNASSIGN 2023-04-07 01:55:48,857 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:48,857 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832548857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832548857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832548857"}]},"ts":"1680832548857"} 2023-04-07 01:55:48,858 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:49,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 16a7d0fcb00e6e5b8493de38ebf26fec, disabling compactions & flushes 2023-04-07 01:55:49,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. after waiting 0 ms 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:49,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 16a7d0fcb00e6e5b8493de38ebf26fec 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=A 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=B 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a7d0fcb00e6e5b8493de38ebf26fec, store=C 2023-04-07 01:55:49,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:49,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040762e92b67cd674d6bbf9d39dae3f70fbc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040762e92b67cd674d6bbf9d39dae3f70fbc 2023-04-07 01:55:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:49,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1268, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/a79a4194bb8043b2a17ea6c43ae79c74 2023-04-07 01:55:49,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:50,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1268 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/b98a4b4ad5a84b5abee9c0df9f456f0d 2023-04-07 01:55:50,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1268 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/5355c976e57d43df8611fb5b449c0c33 2023-04-07 01:55:50,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/A/a79a4194bb8043b2a17ea6c43ae79c74 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/a79a4194bb8043b2a17ea6c43ae79c74 2023-04-07 01:55:50,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/a79a4194bb8043b2a17ea6c43ae79c74, entries=50, sequenceid=1268, filesize=11.7 K 2023-04-07 01:55:50,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/B/b98a4b4ad5a84b5abee9c0df9f456f0d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b98a4b4ad5a84b5abee9c0df9f456f0d 2023-04-07 01:55:50,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b98a4b4ad5a84b5abee9c0df9f456f0d, entries=50, sequenceid=1268, filesize=7.1 K 2023-04-07 01:55:50,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/.tmp/C/5355c976e57d43df8611fb5b449c0c33 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5355c976e57d43df8611fb5b449c0c33 2023-04-07 01:55:50,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5355c976e57d43df8611fb5b449c0c33, entries=50, sequenceid=1268, filesize=7.1 K 2023-04-07 01:55:50,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for 16a7d0fcb00e6e5b8493de38ebf26fec in 1644ms, sequenceid=1268, compaction requested=true 2023-04-07 01:55:50,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits/1271.seqid, newMaxSeqId=1271, maxSeqId=4 2023-04-07 01:55:50,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec. 2023-04-07 01:55:50,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 16a7d0fcb00e6e5b8493de38ebf26fec: 2023-04-07 01:55:50,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:50,663 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=16a7d0fcb00e6e5b8493de38ebf26fec, regionState=CLOSED 2023-04-07 01:55:50,663 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832550662"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832550662"}]},"ts":"1680832550662"} 2023-04-07 01:55:50,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-07 01:55:50,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 16a7d0fcb00e6e5b8493de38ebf26fec, server=jenkins-hbase4.apache.org,44961,1680832421898 in 1.8050 sec 2023-04-07 01:55:50,666 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-07 01:55:50,666 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a7d0fcb00e6e5b8493de38ebf26fec, UNASSIGN in 1.8100 sec 2023-04-07 01:55:50,666 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832550666"}]},"ts":"1680832550666"} 2023-04-07 01:55:50,667 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:55:50,671 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:55:50,673 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8250 sec 2023-04-07 01:55:50,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 01:55:50,957 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-07 01:55:50,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:55:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:50,959 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-07 01:55:50,960 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:50,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:50,965 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits] 2023-04-07 01:55:50,967 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/16a7a47ef0014d478c6ba55e52258383 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/16a7a47ef0014d478c6ba55e52258383 2023-04-07 01:55:50,968 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/518aad10485a4931b5c63eea4dc65839 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/518aad10485a4931b5c63eea4dc65839 2023-04-07 01:55:50,969 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/a79a4194bb8043b2a17ea6c43ae79c74 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/A/a79a4194bb8043b2a17ea6c43ae79c74 2023-04-07 01:55:50,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/08411017d27b43bbb588d03fa949bd94 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/08411017d27b43bbb588d03fa949bd94 2023-04-07 01:55:50,972 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8f2fe24ab6f244ec845f587a1b01e972 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/8f2fe24ab6f244ec845f587a1b01e972 2023-04-07 01:55:50,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b98a4b4ad5a84b5abee9c0df9f456f0d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/B/b98a4b4ad5a84b5abee9c0df9f456f0d 2023-04-07 01:55:50,974 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/3a37445f325748cfa8349f97edd805bb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/3a37445f325748cfa8349f97edd805bb 2023-04-07 01:55:50,975 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5355c976e57d43df8611fb5b449c0c33 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/5355c976e57d43df8611fb5b449c0c33 2023-04-07 01:55:50,976 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/83fdd194fed94ed3bf21dceb1f2a09b6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/C/83fdd194fed94ed3bf21dceb1f2a09b6 2023-04-07 01:55:50,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits/1271.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec/recovered.edits/1271.seqid 2023-04-07 01:55:50,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/16a7d0fcb00e6e5b8493de38ebf26fec 2023-04-07 01:55:50,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:55:50,979 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:55:50,979 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 01:55:50,981 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070253b0c737ba458dad22268a7d131103 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070253b0c737ba458dad22268a7d131103 2023-04-07 01:55:50,982 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407055fa34c22ce46efaa8b74ffda52749b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407055fa34c22ce46efaa8b74ffda52749b 2023-04-07 01:55:50,983 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040706f00a11cfe440adb73953a5df2d1262 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040706f00a11cfe440adb73953a5df2d1262 2023-04-07 01:55:50,984 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072b8a0d1df6274324a233ba95674825f3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072b8a0d1df6274324a233ba95674825f3 2023-04-07 01:55:50,985 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e3a69d6c63344ad9e81dd3bb9f8027e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e3a69d6c63344ad9e81dd3bb9f8027e 2023-04-07 01:55:50,986 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040733adb6fbf9fe47c18e6d73e9ab1f1d82 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040733adb6fbf9fe47c18e6d73e9ab1f1d82 2023-04-07 01:55:50,986 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073a1a7981660943b7b24b7e0a0a18b6fb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073a1a7981660943b7b24b7e0a0a18b6fb 2023-04-07 01:55:50,987 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407459ae852d681433fb5721f1cfb69137c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407459ae852d681433fb5721f1cfb69137c 2023-04-07 01:55:50,988 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075488798dcebb4c4199c55d57811cb949 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075488798dcebb4c4199c55d57811cb949 2023-04-07 01:55:50,989 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040762e92b67cd674d6bbf9d39dae3f70fbc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040762e92b67cd674d6bbf9d39dae3f70fbc 2023-04-07 01:55:50,990 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040763bb77565d324c959aa4a96fa8ef7768 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040763bb77565d324c959aa4a96fa8ef7768 2023-04-07 01:55:50,990 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040778854a4986e14b30b60a45af8df90cdc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040778854a4986e14b30b60a45af8df90cdc 2023-04-07 01:55:50,991 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077b37f39d92df4757bba5273142bc2d73 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077b37f39d92df4757bba5273142bc2d73 2023-04-07 01:55:50,992 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040794b8f4184f974f6493387d4cea9d9585 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040794b8f4184f974f6493387d4cea9d9585 2023-04-07 01:55:50,993 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079a60ce0f87374ed5b2f6b5c41537f7ac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079a60ce0f87374ed5b2f6b5c41537f7ac 2023-04-07 01:55:50,994 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a914eaad66324244b8d216387b128613 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a914eaad66324244b8d216387b128613 2023-04-07 01:55:50,994 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bd8553c8161149dea102efe9972245ac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bd8553c8161149dea102efe9972245ac 2023-04-07 01:55:50,995 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ca6990375b6b468b9af054c6da06af50 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ca6990375b6b468b9af054c6da06af50 2023-04-07 01:55:50,996 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da49c76d086a4272a784c0077973aa4b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da49c76d086a4272a784c0077973aa4b 2023-04-07 01:55:50,997 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e56b9a9aa7f3465a9149de973d851509 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e56b9a9aa7f3465a9149de973d851509 2023-04-07 01:55:50,998 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f5bd268bbc4048a1bbd5228a3da79943 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f5bd268bbc4048a1bbd5228a3da79943 2023-04-07 01:55:50,998 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:55:51,000 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:51,001 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:55:51,002 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:55:51,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:51,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:55:51,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832551003"}]},"ts":"9223372036854775807"} 2023-04-07 01:55:51,004 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:55:51,004 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 16a7d0fcb00e6e5b8493de38ebf26fec, NAME => 'TestAcidGuarantees,,1680832518294.16a7d0fcb00e6e5b8493de38ebf26fec.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:55:51,004 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:55:51,004 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832551004"}]},"ts":"9223372036854775807"} 2023-04-07 01:55:51,005 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:55:51,007 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:51,008 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 50 msec 2023-04-07 01:55:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-07 01:55:51,060 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-07 01:55:51,072 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=547 (was 539) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=300 (was 325), ProcessCount=169 (was 169), AvailableMemoryMB=2509 (was 2548) 2023-04-07 01:55:51,082 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=169, AvailableMemoryMB=2509 2023-04-07 01:55:51,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:55:51,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:55:51,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:55:51,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:55:51,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-07 01:55:51,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:55:51,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 01:55:51,088 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 empty. 2023-04-07 01:55:51,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,089 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:55:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 01:55:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 01:55:51,496 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:55:51,498 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0e0b9000c3ed0a157bd0554455c64d06, NAME => 'TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0e0b9000c3ed0a157bd0554455c64d06, disabling compactions & flushes 2023-04-07 01:55:51,506 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. after waiting 0 ms 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,506 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:51,508 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:55:51,509 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832551508"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832551508"}]},"ts":"1680832551508"} 2023-04-07 01:55:51,509 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:55:51,510 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:55:51,510 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832551510"}]},"ts":"1680832551510"} 2023-04-07 01:55:51,511 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:55:51,514 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, ASSIGN}] 2023-04-07 01:55:51,515 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, ASSIGN 2023-04-07 01:55:51,516 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:55:51,667 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0e0b9000c3ed0a157bd0554455c64d06, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:51,667 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832551667"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832551667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832551667"}]},"ts":"1680832551667"} 2023-04-07 01:55:51,668 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:55:51,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 01:55:51,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0e0b9000c3ed0a157bd0554455c64d06, NAME => 'TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:55:51,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:55:51,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,824 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,827 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A 2023-04-07 01:55:51,827 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A 2023-04-07 01:55:51,827 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:51,827 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e0b9000c3ed0a157bd0554455c64d06 columnFamilyName A 2023-04-07 01:55:51,827 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(310): Store=0e0b9000c3ed0a157bd0554455c64d06/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:51,828 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,828 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B 2023-04-07 01:55:51,829 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B 2023-04-07 01:55:51,829 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:51,829 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e0b9000c3ed0a157bd0554455c64d06 columnFamilyName B 2023-04-07 01:55:51,829 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(310): Store=0e0b9000c3ed0a157bd0554455c64d06/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:51,829 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,830 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C 2023-04-07 01:55:51,830 DEBUG [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C 2023-04-07 01:55:51,830 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:55:51,830 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0e0b9000c3ed0a157bd0554455c64d06 columnFamilyName C 2023-04-07 01:55:51,831 INFO [StoreOpener-0e0b9000c3ed0a157bd0554455c64d06-1] regionserver.HStore(310): Store=0e0b9000c3ed0a157bd0554455c64d06/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:55:51,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:55:51,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:51,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:55:51,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0e0b9000c3ed0a157bd0554455c64d06; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74589951, jitterRate=0.11147688329219818}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:55:51,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:51,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., pid=49, masterSystemTime=1680832551819 2023-04-07 01:55:51,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:51,838 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0e0b9000c3ed0a157bd0554455c64d06, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:51,838 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832551838"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832551838"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832551838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832551838"}]},"ts":"1680832551838"} 2023-04-07 01:55:51,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-07 01:55:51,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 in 171 msec 2023-04-07 01:55:51,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-07 01:55:51,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, ASSIGN in 326 msec 2023-04-07 01:55:51,841 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:55:51,841 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832551841"}]},"ts":"1680832551841"} 2023-04-07 01:55:51,842 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:55:51,847 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:55:51,847 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 762 msec 2023-04-07 01:55:52,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 01:55:52,191 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-07 01:55:52,192 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2a987907 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,196 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2df63a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,197 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,199 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,200 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:55:52,202 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:55:52,203 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,206 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,206 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,210 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,211 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,220 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,220 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,224 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,224 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,227 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,227 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,231 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,231 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,236 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,237 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,240 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,240 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,244 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,244 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:55:52,248 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:55:52,257 DEBUG [hconnection-0x4de22021-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,257 DEBUG [hconnection-0x67652241-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,257 DEBUG [hconnection-0x3c4e2b5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,257 DEBUG [hconnection-0x16bdb542-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,258 DEBUG [hconnection-0x5be18729-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,259 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,260 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,260 DEBUG [hconnection-0x6944eae5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,260 DEBUG [hconnection-0x25edeefb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,266 DEBUG [hconnection-0x136fd94e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,266 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,266 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,266 DEBUG [hconnection-0x7238d4fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,266 DEBUG [hconnection-0x2f9e176d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:55:52,266 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,272 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,273 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,274 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,275 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,277 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:55:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,278 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:52,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:52,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,288 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,288 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,288 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,288 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:52,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,290 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c379dd[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,292 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,292 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,294 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,295 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,295 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,295 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,295 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,301 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,301 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,303 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,303 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,303 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832612301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832612302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832612303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,308 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,308 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832612303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832612304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,408 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:52,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:52,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832612405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832612408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832612409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832612409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832612409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,414 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:52,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,419 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,419 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,419 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,419 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54089226[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,424 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,429 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,429 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,431 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,435 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,435 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,535 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:52,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:52,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,539 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:52,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,543 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,543 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,543 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,543 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,545 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21224d7f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,553 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,554 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,554 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,560 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,565 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,565 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832612610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832612611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832612611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832612611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832612612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,665 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:52,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:52,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,669 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,671 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:52,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,672 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,672 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,672 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,676 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c1e9ab[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,680 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,681 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,684 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,684 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,684 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,690 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,691 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,691 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/23d82d8fb1e44a62816d184453d887f5 2023-04-07 01:55:52,791 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:52,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,794 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,798 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,799 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,799 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,799 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,799 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,801 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423e63ff[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,808 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,809 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,809 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:52,820 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,820 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,920 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:52,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832612921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832612921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832612921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:52,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832612921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:52,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:52,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:52,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:52,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:52,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832612921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,925 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,926 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:52,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:52,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:52,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:52,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:52,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:52,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b7503c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:52,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:52,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:52,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:52,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:52,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:52,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:52,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:52,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:52,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:52,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:52,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:52,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:52,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:52,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:52,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:52,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:52,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:52,944 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:52,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:52,944 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:52,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,044 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,047 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,049 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@614da6a0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,057 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,059 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,066 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,067 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,067 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d995079b56f04b4fa741acada230381c 2023-04-07 01:55:53,167 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,170 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,170 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,173 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,175 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,175 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,175 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:53,175 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,177 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7e8793[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,179 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,183 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,184 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:53,194 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,194 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,294 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,297 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,299 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,299 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,299 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,299 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:53,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,302 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fee3638[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,305 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,310 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,310 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,313 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,315 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,316 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,316 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:53,416 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,419 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,422 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,422 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,422 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:53,423 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832613423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730ad49[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:53,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832613424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832613425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832613427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,432 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:53,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832613428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,435 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,435 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,437 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,440 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,446 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,446 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,446 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f18e1bacf90a470faa7e79575aea410a 2023-04-07 01:55:53,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/23d82d8fb1e44a62816d184453d887f5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5 2023-04-07 01:55:53,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5, entries=200, sequenceid=14, filesize=13.9 K 2023-04-07 01:55:53,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d995079b56f04b4fa741acada230381c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c 2023-04-07 01:55:53,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c, entries=150, sequenceid=14, filesize=11.6 K 2023-04-07 01:55:53,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f18e1bacf90a470faa7e79575aea410a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a 2023-04-07 01:55:53,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a, entries=150, sequenceid=14, filesize=11.6 K 2023-04-07 01:55:53,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0e0b9000c3ed0a157bd0554455c64d06 in 1254ms, sequenceid=14, compaction requested=false 2023-04-07 01:55:53,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:53,546 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,549 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,553 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:53,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:53,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:53,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:53,564 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9286ffad180e40c49d13a80077b4cad4 2023-04-07 01:55:53,581 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0da6d0a7868940d69d5ac4b461f79265 2023-04-07 01:55:53,592 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3805ff39524b4da584a1eada245530e4 2023-04-07 01:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9286ffad180e40c49d13a80077b4cad4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4 2023-04-07 01:55:53,598 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 01:55:53,599 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0da6d0a7868940d69d5ac4b461f79265 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265 2023-04-07 01:55:53,606 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 01:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3805ff39524b4da584a1eada245530e4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4 2023-04-07 01:55:53,610 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 01:55:53,610 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 57ms, sequenceid=33, compaction requested=false 2023-04-07 01:55:53,610 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:53,610 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc473a9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,623 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,623 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,623 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,624 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,625 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,626 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:53,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,630 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,631 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,731 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,735 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,735 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,739 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,744 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,744 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,744 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:53,744 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,746 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212c3f5d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,749 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,752 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,754 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,754 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,754 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,757 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,757 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,757 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,758 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,763 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,763 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,762 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,763 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,863 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,866 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,868 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:53,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:53,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:53,870 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,870 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:53,870 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:53,870 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,872 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,872 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:53,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fed92b6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:53,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:53,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:53,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,877 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:53,878 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:53,883 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:53,883 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:53,883 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:53,883 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:53,883 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:53,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:53,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,887 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,888 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:53,890 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:53,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:53,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:53,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:53,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:53,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:53,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:53,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:53,894 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:53,894 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:53,994 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:53,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:53,998 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:53,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:53,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:53,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:53,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:53,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,000 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,001 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,001 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,003 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,003 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,003 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:54,004 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d353ae7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,010 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,012 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,014 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,014 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,017 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,019 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,021 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,021 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,121 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,126 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,128 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:54,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,135 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,135 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c45fd3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,137 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,139 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,140 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,140 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,141 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,141 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,144 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,146 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,146 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,152 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,152 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,152 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,152 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,152 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,152 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,252 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,256 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,257 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,257 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,258 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,259 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,259 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:54,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,261 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55524ce1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,263 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,264 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,265 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,268 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,271 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,271 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,280 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,280 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,281 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,381 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,384 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,385 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,386 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:54,389 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,392 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f90cdf[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,400 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,403 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,404 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,407 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,415 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,415 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:54,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,416 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:54,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:54,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832554454, max timestamp=1680832554471Num uniques 859; 2023-04-07 01:55:54,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832554454, max timestamp=1680832554471Num uniques 859; 2023-04-07 01:55:54,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:54,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:54,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832554454, max timestamp=1680832554471Num uniques 859; 2023-04-07 01:55:54,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:55:54,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832614494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832614494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832614501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832614501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832614504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,515 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,522 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,522 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,523 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,524 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,536 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,536 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,536 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:54,536 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,538 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128fece5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,548 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,553 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,553 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,567 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,567 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832614605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832614605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832614613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832614614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832614614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,667 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,670 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,670 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,672 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,676 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@250203a0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,684 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,684 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,689 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,691 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,691 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,792 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,795 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,797 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:54,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:54,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627076a5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:54,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:54,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:54,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:54,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:54,812 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:54,812 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:54,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832614812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:54,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:54,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832614813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:54,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,819 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:54,819 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:54,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:54,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:54,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:54,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832614819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832614820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832614820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c1a3d0895d85406c9bc2659f3b0aa192 2023-04-07 01:55:54,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/caa3974421a94421a7634444f738f7c0 2023-04-07 01:55:54,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/a38fcac604404274972f728933d9444d 2023-04-07 01:55:54,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c1a3d0895d85406c9bc2659f3b0aa192 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192 2023-04-07 01:55:54,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192, entries=300, sequenceid=49, filesize=18.5 K 2023-04-07 01:55:54,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/caa3974421a94421a7634444f738f7c0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0 2023-04-07 01:55:54,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0, entries=150, sequenceid=49, filesize=11.6 K 2023-04-07 01:55:54,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/a38fcac604404274972f728933d9444d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d 2023-04-07 01:55:54,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d, entries=150, sequenceid=49, filesize=11.6 K 2023-04-07 01:55:54,888 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 01:55:54,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 0e0b9000c3ed0a157bd0554455c64d06 in 453ms, sequenceid=49, compaction requested=true 2023-04-07 01:55:54,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:54,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:54,890 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:54,890 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:54,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:54,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:54,891 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45018 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:54,891 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:55:54,891 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:54,891 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=44.0 K 2023-04-07 01:55:54,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:55:54,891 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,891 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.8 K 2023-04-07 01:55:54,891 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 23d82d8fb1e44a62816d184453d887f5, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832552278 2023-04-07 01:55:54,892 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d995079b56f04b4fa741acada230381c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832552278 2023-04-07 01:55:54,892 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9286ffad180e40c49d13a80077b4cad4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680832552289 2023-04-07 01:55:54,892 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0da6d0a7868940d69d5ac4b461f79265, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680832552289 2023-04-07 01:55:54,892 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c1a3d0895d85406c9bc2659f3b0aa192, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:54,892 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting caa3974421a94421a7634444f738f7c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:54,897 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#426 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:54,898 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:54,919 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:54,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:54,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:54,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:54,922 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:54,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:54,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:54,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,924 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:54,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:54,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:54,926 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:54,926 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:54,934 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/d86511f998c247af8f68754e7d181069 2023-04-07 01:55:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:55,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:55,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832615144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832615144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832615145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832615145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832615145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832615258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832615259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832615260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832615260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832615260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,306 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0f1e5b66a25148e195d5fc74ac3790f4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0f1e5b66a25148e195d5fc74ac3790f4 2023-04-07 01:55:55,312 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 0f1e5b66a25148e195d5fc74ac3790f4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:55,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:55,312 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832554890; duration=0sec 2023-04-07 01:55:55,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:55,312 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:55,314 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:55,314 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:55:55,314 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,314 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.8 K 2023-04-07 01:55:55,314 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f18e1bacf90a470faa7e79575aea410a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680832552278 2023-04-07 01:55:55,315 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3805ff39524b4da584a1eada245530e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680832552289 2023-04-07 01:55:55,315 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a38fcac604404274972f728933d9444d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:55,316 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b3e860719d144f0d8f28e8f7b270fd82 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b3e860719d144f0d8f28e8f7b270fd82 2023-04-07 01:55:55,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into b3e860719d144f0d8f28e8f7b270fd82(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:55,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:55,321 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832554890; duration=0sec 2023-04-07 01:55:55,321 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:55,321 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:55,328 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/272289ccc6ba4f8cbb89923ca0066ebf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/272289ccc6ba4f8cbb89923ca0066ebf 2023-04-07 01:55:55,332 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 272289ccc6ba4f8cbb89923ca0066ebf(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:55,332 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:55,332 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832554890; duration=0sec 2023-04-07 01:55:55,332 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:55,345 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/989ab59ff40a4888990526e968ebb5a6 2023-04-07 01:55:55,363 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/d6bec5b80dec4859ab92223128813137 2023-04-07 01:55:55,367 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/d86511f998c247af8f68754e7d181069 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069 2023-04-07 01:55:55,373 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069, entries=150, sequenceid=84, filesize=11.6 K 2023-04-07 01:55:55,374 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/989ab59ff40a4888990526e968ebb5a6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6 2023-04-07 01:55:55,378 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6, entries=150, sequenceid=84, filesize=11.6 K 2023-04-07 01:55:55,378 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/d6bec5b80dec4859ab92223128813137 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137 2023-04-07 01:55:55,395 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137, entries=150, sequenceid=84, filesize=11.6 K 2023-04-07 01:55:55,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for 0e0b9000c3ed0a157bd0554455c64d06 in 470ms, sequenceid=84, compaction requested=false 2023-04-07 01:55:55,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:55,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:55,400 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2111c0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:55,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:55,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,404 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:55,404 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:55,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:55,410 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:55,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:55,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:55,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:55,410 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:55,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,418 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:55,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:55,421 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:55,421 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:55,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:55,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:55,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:55,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:55,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:55,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:55,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:55,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:55,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832615484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832615485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832615486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832615487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832615487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,521 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:55,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:55,525 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:55,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:55,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:55,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,528 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:55,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:55,530 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,530 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:55,530 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:55,530 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:55,532 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41242865[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:55,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:55,533 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:55,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:55,538 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:55,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:55,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:55,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:55,538 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:55,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:55,545 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:55,545 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832615590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832615594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832615595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832615595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832615595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,645 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:55,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:55,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:55,649 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,650 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:55,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:55,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:55,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:55,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:55,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:55,660 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,660 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740e3ce2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:55,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:55,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,671 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:55,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:55,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,673 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:55,673 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,673 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:55,676 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:55,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:55,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:55,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:55,676 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:55,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,687 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:55,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:55,692 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:55,692 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:55,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:55,792 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:55,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:55,795 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:55,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:55,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:55,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832615792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,798 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:55,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:55,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:55,801 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:55,801 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:55,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:55,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832615799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832615800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832615800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:55,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152aa998[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:55,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:55,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,806 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:55,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:55,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:55,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:55,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:55,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832615801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:55,811 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:55,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:55,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:55,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:55,811 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:55,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:55,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:55,826 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:55,826 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:55,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/07386df652284fe7ac3d432745595a01 2023-04-07 01:55:55,926 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:55,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:55,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:55,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:55,930 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:55,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:55,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:55,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:55,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:55,934 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,934 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:55,934 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:55,934 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:55,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:55,937 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312105ec[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:55,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:55,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,941 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:55,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:55,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:55,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,943 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:55,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:55,946 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:55,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:55,946 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:55,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:55,948 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:55,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:55,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:55,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:55,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:55,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,953 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:55,953 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:55,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:55,953 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,053 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,057 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,059 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,060 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512664ad[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,070 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,070 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,077 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,077 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,077 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832616099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832616106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832616106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832616107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832616109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,177 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,181 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,183 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,183 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54caab47[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,196 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,197 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,198 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:55:56,199 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,201 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:56,212 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,212 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/acf21b587764411e8dcae548e890a738 2023-04-07 01:55:56,312 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,323 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,324 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146cae2f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,339 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,339 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,339 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,341 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,346 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,346 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,346 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,346 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,446 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,449 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,451 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,453 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,454 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,458 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,459 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5bd72b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,461 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,462 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,462 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,464 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,467 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,467 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,468 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,469 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,470 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,474 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,474 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:56,574 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,578 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,579 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,579 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,580 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,580 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,581 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,581 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,583 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,583 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3acd21[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,591 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,594 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,594 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,598 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,601 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,602 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,602 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832616606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832616610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832616611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832616614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:56,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832616615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,702 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/33036ab51de443b288e5edbeb4cf46a0 2023-04-07 01:55:56,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,705 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,706 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,706 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,707 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,708 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,708 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,709 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,709 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,709 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:56,709 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/07386df652284fe7ac3d432745595a01 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01 2023-04-07 01:55:56,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01, entries=250, sequenceid=100, filesize=16.2 K 2023-04-07 01:55:56,714 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:56,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/acf21b587764411e8dcae548e890a738 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738 2023-04-07 01:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eabd712[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:56,717 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:56,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738, entries=150, sequenceid=100, filesize=11.6 K 2023-04-07 01:55:56,719 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:56,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/33036ab51de443b288e5edbeb4cf46a0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0 2023-04-07 01:55:56,720 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:56,721 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,721 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:56,721 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:56,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:56,721 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:56,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:56,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:56,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:56,729 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,729 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,729 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:56,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0, entries=150, sequenceid=100, filesize=11.6 K 2023-04-07 01:55:56,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0e0b9000c3ed0a157bd0554455c64d06 in 1265ms, sequenceid=100, compaction requested=true 2023-04-07 01:55:56,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:56,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:56,730 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:56,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:56,730 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:56,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:56,732 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:56,732 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:56,733 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:55:56,733 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:55:56,733 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,733 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,733 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0f1e5b66a25148e195d5fc74ac3790f4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.9 K 2023-04-07 01:55:56,733 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b3e860719d144f0d8f28e8f7b270fd82, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=39.5 K 2023-04-07 01:55:56,733 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0f1e5b66a25148e195d5fc74ac3790f4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:56,733 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b3e860719d144f0d8f28e8f7b270fd82, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:56,735 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 989ab59ff40a4888990526e968ebb5a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680832554493 2023-04-07 01:55:56,735 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d86511f998c247af8f68754e7d181069, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680832554493 2023-04-07 01:55:56,736 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting acf21b587764411e8dcae548e890a738, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555143 2023-04-07 01:55:56,736 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 07386df652284fe7ac3d432745595a01, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555131 2023-04-07 01:55:56,745 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:56,745 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:56,758 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/8405f54396654abcb2d837405acffc63 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8405f54396654abcb2d837405acffc63 2023-04-07 01:55:56,759 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/54182927106d407ab25b19b973b9814a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/54182927106d407ab25b19b973b9814a 2023-04-07 01:55:56,766 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 8405f54396654abcb2d837405acffc63(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:56,766 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:56,766 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832556730; duration=0sec 2023-04-07 01:55:56,766 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:56,766 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:56,774 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:56,774 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:55:56,774 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,774 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/272289ccc6ba4f8cbb89923ca0066ebf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.9 K 2023-04-07 01:55:56,776 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 272289ccc6ba4f8cbb89923ca0066ebf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680832554430 2023-04-07 01:55:56,776 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 54182927106d407ab25b19b973b9814a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:56,776 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:56,776 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832556730; duration=0sec 2023-04-07 01:55:56,776 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:56,777 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d6bec5b80dec4859ab92223128813137, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680832554493 2023-04-07 01:55:56,777 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 33036ab51de443b288e5edbeb4cf46a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555143 2023-04-07 01:55:56,785 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:56,801 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/2c1d301a742f472fa68e8cc73c573d64 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2c1d301a742f472fa68e8cc73c573d64 2023-04-07 01:55:56,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 2c1d301a742f472fa68e8cc73c573d64(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:56,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:56,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832556730; duration=0sec 2023-04-07 01:55:56,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:56,830 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:56,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:56,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:56,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:56,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:56,835 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:56,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:56,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:56,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,836 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:56,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:56,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:56,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:56,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,840 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:56,841 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:56,841 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:56,841 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,841 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:56,842 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:56,842 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:56,859 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/2ac7e33cf606450f9733e73f18c1750f 2023-04-07 01:55:56,932 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/ec6cb31d8ff946d28b24947825c410c0 2023-04-07 01:55:56,968 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f530c3ecae174e40ae30645bdc4c0500 2023-04-07 01:55:56,973 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/2ac7e33cf606450f9733e73f18c1750f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f 2023-04-07 01:55:56,978 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f, entries=150, sequenceid=122, filesize=11.6 K 2023-04-07 01:55:56,978 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/ec6cb31d8ff946d28b24947825c410c0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0 2023-04-07 01:55:56,984 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0, entries=150, sequenceid=122, filesize=11.6 K 2023-04-07 01:55:56,985 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f530c3ecae174e40ae30645bdc4c0500 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500 2023-04-07 01:55:56,993 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500, entries=150, sequenceid=122, filesize=11.6 K 2023-04-07 01:55:56,994 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 151ms, sequenceid=122, compaction requested=false 2023-04-07 01:55:56,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:56,994 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:56,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f47053e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,001 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,001 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,002 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,002 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,003 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,005 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,009 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,009 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,010 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,011 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,017 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,018 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,018 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,018 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,018 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,118 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,122 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,123 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,123 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,127 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd66f72[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,129 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,130 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,135 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,135 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,137 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,141 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,141 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,141 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,241 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,244 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:57,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,251 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559cf00c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,254 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,255 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,261 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,267 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,267 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,277 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,282 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,283 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,283 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,383 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,386 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,389 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:57,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,394 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451faeea[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,400 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,406 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,406 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,411 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,415 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,415 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,515 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,518 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,520 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:57,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,524 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42762d82[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,528 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,532 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,532 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,533 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,533 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,533 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,537 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,538 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,538 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:57,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:57,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:57,638 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,641 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,644 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,646 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,646 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,646 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:57,646 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,647 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,647 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d246741[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,652 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/6a0d35876dc242f08a425cdabc45068c 2023-04-07 01:55:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,657 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,657 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,657 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,657 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,657 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680832557637, max timestamp=1680832557657Num uniques 859; 2023-04-07 01:55:57,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-07 01:55:57,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680832557637, max timestamp=1680832557657Num uniques 859; 2023-04-07 01:55:57,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-07 01:55:57,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680832557637, max timestamp=1680832557657Num uniques 859; 2023-04-07 01:55:57,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-07 01:55:57,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,665 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/319b1879bb51494cbabf7cb16dbc51be 2023-04-07 01:55:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,674 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,674 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,673 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832617670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832617672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832617671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832617675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832617684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,774 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,777 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,781 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,781 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,781 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:57,781 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,783 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,783 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8aec79[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832617784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832617785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832617785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,808 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,808 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832617786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:57,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832617788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,811 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,815 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,815 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,915 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:57,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:57,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:57,918 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:57,918 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:57,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,925 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,925 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:57,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:57,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:57,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:57,927 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:57,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,929 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37da6b9c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:57,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:57,930 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:57,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:57,933 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:57,935 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:57,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:57,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:57,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:57,936 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:57,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,939 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:57,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:57,942 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:57,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:57,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:57,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:57,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:57,943 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:57,943 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:57,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:57,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:57,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:57,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:57,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:57,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832618006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832618008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832618010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832618010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832618010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,043 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:58,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:58,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:58,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:58,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:58,046 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:58,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:58,046 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:58,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:58,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:58,049 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:58,050 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,050 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:58,050 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:58,050 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:58,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b35ad8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:58,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:58,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:58,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:58,057 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:58,060 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:58,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:58,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:58,060 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:58,060 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:58,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:58,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:58,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,071 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:58,071 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7508e4fe6e224d80937f0399de6640fd 2023-04-07 01:55:58,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/6a0d35876dc242f08a425cdabc45068c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c 2023-04-07 01:55:58,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c, entries=200, sequenceid=137, filesize=14.1 K 2023-04-07 01:55:58,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/319b1879bb51494cbabf7cb16dbc51be as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be 2023-04-07 01:55:58,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be, entries=150, sequenceid=137, filesize=11.8 K 2023-04-07 01:55:58,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7508e4fe6e224d80937f0399de6640fd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd 2023-04-07 01:55:58,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd, entries=150, sequenceid=137, filesize=11.8 K 2023-04-07 01:55:58,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 0e0b9000c3ed0a157bd0554455c64d06 in 466ms, sequenceid=137, compaction requested=true 2023-04-07 01:55:58,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:58,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:58,093 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:58,093 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:58,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:58,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:58,094 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38404 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:58,094 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:55:58,094 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,094 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8405f54396654abcb2d837405acffc63, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=37.5 K 2023-04-07 01:55:58,094 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8405f54396654abcb2d837405acffc63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555143 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:55:58,095 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,095 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/54182927106d407ab25b19b973b9814a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.2 K 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2ac7e33cf606450f9733e73f18c1750f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680832555484 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 54182927106d407ab25b19b973b9814a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555143 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6a0d35876dc242f08a425cdabc45068c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:55:58,095 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ec6cb31d8ff946d28b24947825c410c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680832555484 2023-04-07 01:55:58,096 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 319b1879bb51494cbabf7cb16dbc51be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:55:58,102 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#443 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:58,102 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:58,171 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:58,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:58,174 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,176 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,177 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:58,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:58,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:58,178 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,178 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:58,178 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-04-07 01:55:58,178 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:58,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:58,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:58,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:58,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:58,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:58,190 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/ddb78f730fb04e26be8bc4efe8328348 2023-04-07 01:55:58,203 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/40cb428a8ed148c7a8a41008fcf35613 2023-04-07 01:55:58,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:58,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832618326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832618326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832618327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832618330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832618339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832618440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832618440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832618440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832618443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832618443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,513 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/80ad28af6a5446259f8c8e5a077a1e27 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/80ad28af6a5446259f8c8e5a077a1e27 2023-04-07 01:55:58,517 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 80ad28af6a5446259f8c8e5a077a1e27(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:58,518 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:58,518 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832558093; duration=0sec 2023-04-07 01:55:58,518 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:58,518 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:58,519 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:58,519 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:55:58,519 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/fb748c2af82344b19d75059db122bc96 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/fb748c2af82344b19d75059db122bc96 2023-04-07 01:55:58,519 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,520 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2c1d301a742f472fa68e8cc73c573d64, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.2 K 2023-04-07 01:55:58,521 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2c1d301a742f472fa68e8cc73c573d64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832555143 2023-04-07 01:55:58,521 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f530c3ecae174e40ae30645bdc4c0500, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680832555484 2023-04-07 01:55:58,522 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7508e4fe6e224d80937f0399de6640fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:55:58,525 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into fb748c2af82344b19d75059db122bc96(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:58,525 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:58,525 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832558093; duration=0sec 2023-04-07 01:55:58,525 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:58,528 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:58,616 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3e6e171b289d4431a92ba169e187ce13 2023-04-07 01:55:58,621 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/ddb78f730fb04e26be8bc4efe8328348 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348 2023-04-07 01:55:58,625 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348, entries=150, sequenceid=169, filesize=11.8 K 2023-04-07 01:55:58,625 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/40cb428a8ed148c7a8a41008fcf35613 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613 2023-04-07 01:55:58,646 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613, entries=150, sequenceid=169, filesize=11.8 K 2023-04-07 01:55:58,647 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3e6e171b289d4431a92ba169e187ce13 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13 2023-04-07 01:55:58,654 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13, entries=150, sequenceid=169, filesize=11.8 K 2023-04-07 01:55:58,655 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=53.67 KB/54960 for 0e0b9000c3ed0a157bd0554455c64d06 in 477ms, sequenceid=169, compaction requested=false 2023-04-07 01:55:58,655 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:58,655 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:58,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:55:58,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:58,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:58,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:58,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:58,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:58,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:55:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:58,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab33360[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:58,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:58,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,663 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:58,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:58,666 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:58,669 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:58,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:58,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:58,669 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:58,669 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:58,671 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,671 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,671 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:58,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:58,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,678 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:58,678 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:58,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832618687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7274, totalHeapSize=16776, min timestamp=1680832558689, max timestamp=1680832558695Num uniques 859; 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7274, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7274, totalHeapSize=16776, min timestamp=1680832558689, max timestamp=1680832558695Num uniques 859; 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7274, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7274, totalHeapSize=16776, min timestamp=1680832558689, max timestamp=1680832558695Num uniques 859; 2023-04-07 01:55:58,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7274, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:55:58,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832618732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832618732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832618733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832618733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,778 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:58,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:58,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:58,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:58,782 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:58,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:58,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:58,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,783 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:58,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:58,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:58,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:58,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:58,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:58,786 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:58,787 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,787 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:58,787 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:58,789 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282e56b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:58,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:58,791 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,792 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:58,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:58,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:58,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:58,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:58,794 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:58,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:58,794 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:58,795 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:58,795 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:58,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,798 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832618799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:58,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:58,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,808 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:58,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,808 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:58,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,809 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832618842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832618842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832618843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832618843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,909 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:58,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:58,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:58,912 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:58,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:58,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:58,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:58,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:58,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:58,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:58,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:58,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:58,918 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,918 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,919 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,919 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6660e82c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:58,920 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:58,920 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:58,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:58,926 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:58,928 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:58,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:58,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:58,929 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:58,929 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:58,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,930 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:58,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:58,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:58,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:58,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:58,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:58,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:58,938 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:58,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:58,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:58,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:58,939 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:58,939 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:58,941 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/ecbefac8940d432c8314dda7c34e70e5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ecbefac8940d432c8314dda7c34e70e5 2023-04-07 01:55:58,945 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into ecbefac8940d432c8314dda7c34e70e5(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:55:58,945 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:58,945 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832558093; duration=0sec 2023-04-07 01:55:58,946 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:59,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832619009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,039 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,042 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,044 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,045 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,045 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,047 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,047 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,047 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,047 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,050 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385b1ba2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,053 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832619049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832619050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832619050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832619050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,062 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,069 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,069 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b2fb33affac94116af122fd23533ea4a 2023-04-07 01:55:59,169 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,172 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,173 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,176 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3007fa[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,179 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,181 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,185 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,185 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,187 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,193 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,193 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,293 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,297 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,300 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,302 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,302 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,302 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,302 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,303 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8c1c4a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,309 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,312 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,312 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,319 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,319 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,319 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,320 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832619319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832619356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832619357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832619357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832619358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,420 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,423 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,429 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,431 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,431 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,431 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,431 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,431 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,435 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fccb83f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,444 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,444 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,456 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,457 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,457 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,457 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0a1257ecb73f4b4e8d67f387857c6622 2023-04-07 01:55:59,557 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,561 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,563 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,570 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,571 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6faa346f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,577 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,578 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,578 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,581 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,585 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,585 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,585 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,588 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,594 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,594 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,594 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,594 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,694 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,698 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,698 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,698 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,698 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,702 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,702 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,702 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,702 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,702 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145ef1b7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,711 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,712 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,713 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,725 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,725 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,734 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,734 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,735 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,735 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,835 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832619828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,841 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,843 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:55:59,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,854 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3ed13a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:55:59,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:55:59,857 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:55:59,862 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:55:59,864 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:55:59,866 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:55:59,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:55:59,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:55:59,866 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:55:59,866 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:55:59,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:55:59,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,869 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:55:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832619869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832619871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832619871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:55:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,874 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:55:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:55:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:55:59,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:55:59,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:55:59,876 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:55:59,876 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:55:59,877 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,877 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:55:59,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:55:59,877 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:55:59,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:55:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:55:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:55:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832619872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7f58ec8af8a0489b9513e179ad41124b 2023-04-07 01:55:59,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b2fb33affac94116af122fd23533ea4a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a 2023-04-07 01:55:59,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a, entries=200, sequenceid=183, filesize=14.1 K 2023-04-07 01:55:59,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0a1257ecb73f4b4e8d67f387857c6622 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622 2023-04-07 01:55:59,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622, entries=150, sequenceid=183, filesize=11.8 K 2023-04-07 01:55:59,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7f58ec8af8a0489b9513e179ad41124b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b 2023-04-07 01:55:59,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b, entries=150, sequenceid=183, filesize=11.8 K 2023-04-07 01:55:59,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 0e0b9000c3ed0a157bd0554455c64d06 in 1268ms, sequenceid=183, compaction requested=true 2023-04-07 01:55:59,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:55:59,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:55:59,927 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:59,927 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:55:59,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:59,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:55:59,928 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:59,928 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:55:59,928 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:55:59,928 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:55:59,928 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,929 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,929 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/80ad28af6a5446259f8c8e5a077a1e27, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=37.9 K 2023-04-07 01:55:59,929 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/fb748c2af82344b19d75059db122bc96, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.6 K 2023-04-07 01:55:59,929 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 80ad28af6a5446259f8c8e5a077a1e27, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:55:59,929 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting fb748c2af82344b19d75059db122bc96, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:55:59,933 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 40cb428a8ed148c7a8a41008fcf35613, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680832557667 2023-04-07 01:55:59,933 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ddb78f730fb04e26be8bc4efe8328348, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680832557667 2023-04-07 01:55:59,934 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0a1257ecb73f4b4e8d67f387857c6622, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680832558325 2023-04-07 01:55:59,934 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b2fb33affac94116af122fd23533ea4a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680832558325 2023-04-07 01:55:59,941 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:59,943 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:55:59,977 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:55:59,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:55:59,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:55:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:55:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:55:59,981 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:55:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:55:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:55:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,983 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,983 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:55:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:55:59,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:55:59,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:55:59,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:55:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:55:59,986 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:55:59,986 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:00,018 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/4a25070e21af427a8275a0d10cacad25 2023-04-07 01:56:00,062 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/c80afae854c6444ab959bf5716487ab3 2023-04-07 01:56:00,096 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/25f5c8ab5800451f90fc235d3300a3f2 2023-04-07 01:56:00,102 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/4a25070e21af427a8275a0d10cacad25 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25 2023-04-07 01:56:00,107 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 01:56:00,108 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/c80afae854c6444ab959bf5716487ab3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3 2023-04-07 01:56:00,118 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 01:56:00,119 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/25f5c8ab5800451f90fc235d3300a3f2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2 2023-04-07 01:56:00,129 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 01:56:00,130 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 144ms, sequenceid=220, compaction requested=true 2023-04-07 01:56:00,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,130 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,130 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,132 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70599f02[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,136 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,137 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,138 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,138 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,138 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,138 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,139 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,143 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,144 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,146 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,146 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,146 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,150 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,150 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,151 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,151 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,251 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,255 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,255 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,256 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,256 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,257 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,257 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,260 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,262 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,262 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,262 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,262 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eeed42a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,264 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,266 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,268 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,271 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,271 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:00,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,279 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,279 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,358 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/a53b248b059c455ba4db131c95e33d54 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/a53b248b059c455ba4db131c95e33d54 2023-04-07 01:56:00,362 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into a53b248b059c455ba4db131c95e33d54(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:00,362 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,362 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832559927; duration=0sec 2023-04-07 01:56:00,362 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:00,362 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:00,364 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:00,364 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:00,364 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,364 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ecbefac8940d432c8314dda7c34e70e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=47.3 K 2023-04-07 01:56:00,365 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ecbefac8940d432c8314dda7c34e70e5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680832557615 2023-04-07 01:56:00,365 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3e6e171b289d4431a92ba169e187ce13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680832557667 2023-04-07 01:56:00,365 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7f58ec8af8a0489b9513e179ad41124b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680832558325 2023-04-07 01:56:00,366 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 25f5c8ab5800451f90fc235d3300a3f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680832558731 2023-04-07 01:56:00,367 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/f3e165fed8804dcab1ce74a36f52096e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/f3e165fed8804dcab1ce74a36f52096e 2023-04-07 01:56:00,371 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into f3e165fed8804dcab1ce74a36f52096e(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:00,371 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,371 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832559927; duration=0sec 2023-04-07 01:56:00,371 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:00,379 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,385 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,387 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:00,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dde5709[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,409 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7bfa18f83cc44bd38c47a53568ee6a3c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7bfa18f83cc44bd38c47a53568ee6a3c 2023-04-07 01:56:00,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,412 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,413 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 7bfa18f83cc44bd38c47a53568ee6a3c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:00,413 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,413 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=12, startTime=1680832559928; duration=0sec 2023-04-07 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,413 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,414 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,414 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,415 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,419 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,419 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,426 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,426 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,526 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,530 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,532 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,534 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,540 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4185e489[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,549 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,549 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,552 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,557 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,561 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,561 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,661 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,665 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,668 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,671 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,671 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,671 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,671 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,671 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d98d234[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,680 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,682 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,682 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,682 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,682 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,682 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,684 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,684 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,684 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,686 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,694 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,695 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,695 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:00,795 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,798 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,799 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,800 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,804 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,804 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,805 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,805 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,808 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456f1629[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,812 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,822 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,823 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,833 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:00,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,833 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,834 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,834 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,834 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:00,834 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:00,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:00,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:00,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/959de7cf822a40acb764e093437e67eb 2023-04-07 01:56:00,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7485ac6afa3d4c6eba98b54b1b2ee40c 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832560889, max timestamp=1680832560898Num uniques 859; 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832560889, max timestamp=1680832560898Num uniques 859; 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832560889, max timestamp=1680832560898Num uniques 859; 2023-04-07 01:56:00,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:00,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/11738473a0464acc8ec4e0cf47676c15 2023-04-07 01:56:00,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/959de7cf822a40acb764e093437e67eb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb 2023-04-07 01:56:00,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 01:56:00,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7485ac6afa3d4c6eba98b54b1b2ee40c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c 2023-04-07 01:56:00,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 01:56:00,934 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:00,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/11738473a0464acc8ec4e0cf47676c15 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15 2023-04-07 01:56:00,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:00,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:00,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:00,937 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:00,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:00,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:00,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 01:56:00,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=114.05 KB/116790 for 0e0b9000c3ed0a157bd0554455c64d06 in 84ms, sequenceid=238, compaction requested=true 2023-04-07 01:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:00,939 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:00,939 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:00,940 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:00,940 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:00,940 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:00,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-07 01:56:00,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:00,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:00,940 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:00,940 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,940 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/f3e165fed8804dcab1ce74a36f52096e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.7 K 2023-04-07 01:56:00,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:00,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:00,940 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,941 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/a53b248b059c455ba4db131c95e33d54, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.7 K 2023-04-07 01:56:00,941 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f3e165fed8804dcab1ce74a36f52096e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680832558325 2023-04-07 01:56:00,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,941 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a53b248b059c455ba4db131c95e33d54, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680832558325 2023-04-07 01:56:00,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,942 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4a25070e21af427a8275a0d10cacad25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680832558731 2023-04-07 01:56:00,942 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c80afae854c6444ab959bf5716487ab3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680832558731 2023-04-07 01:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:00,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:00,942 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 959de7cf822a40acb764e093437e67eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680832560845 2023-04-07 01:56:00,943 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7485ac6afa3d4c6eba98b54b1b2ee40c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680832560845 2023-04-07 01:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:00,953 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#461 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:00,957 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:00,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832620950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:00,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832620953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:00,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832620954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:00,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832620955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:00,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832620955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:00,968 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,968 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:00,968 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:00,968 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:00,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,971 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,971 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:00,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad67792[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:00,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:00,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:00,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:00,980 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:00,991 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:00,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:00,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:00,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:00,991 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:00,993 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,993 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:00,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:00,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,998 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,001 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,001 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,001 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,001 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/f00903e1f3e14044afc366f7c20e7d01 2023-04-07 01:56:01,025 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/60b35d9f1dfb4f369192e2a3119781d2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/60b35d9f1dfb4f369192e2a3119781d2 2023-04-07 01:56:01,029 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 60b35d9f1dfb4f369192e2a3119781d2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,029 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,030 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832560938; duration=0sec 2023-04-07 01:56:01,030 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,030 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:01,030 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:01,031 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:01,031 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:01,046 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/16f16f8060494a67b469d145e175ad87 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/16f16f8060494a67b469d145e175ad87 2023-04-07 01:56:01,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/2c727ac98e3a4b2ea3872e958d582a63 2023-04-07 01:56:01,051 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 16f16f8060494a67b469d145e175ad87(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,051 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,051 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832560939; duration=0sec 2023-04-07 01:56:01,051 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/4419abf9612942a18b4b03e59684a007 2023-04-07 01:56:01,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/f00903e1f3e14044afc366f7c20e7d01 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01 2023-04-07 01:56:01,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832621062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832621065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832621065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832621065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832621065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01, entries=200, sequenceid=272, filesize=14.3 K 2023-04-07 01:56:01,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/2c727ac98e3a4b2ea3872e958d582a63 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63 2023-04-07 01:56:01,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63, entries=150, sequenceid=272, filesize=11.9 K 2023-04-07 01:56:01,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/4419abf9612942a18b4b03e59684a007 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007 2023-04-07 01:56:01,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007, entries=150, sequenceid=272, filesize=11.9 K 2023-04-07 01:56:01,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 0e0b9000c3ed0a157bd0554455c64d06 in 138ms, sequenceid=272, compaction requested=true 2023-04-07 01:56:01,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,078 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,078 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:01,078 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,078 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,078 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:01,078 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:01,078 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:01,079 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:01,079 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,079 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7bfa18f83cc44bd38c47a53568ee6a3c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=35.8 K 2023-04-07 01:56:01,080 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7bfa18f83cc44bd38c47a53568ee6a3c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680832558731 2023-04-07 01:56:01,080 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 11738473a0464acc8ec4e0cf47676c15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680832560845 2023-04-07 01:56:01,080 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4419abf9612942a18b4b03e59684a007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680832560902 2023-04-07 01:56:01,084 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,101 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,104 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/526833cf71c141eaa83b713c8df6d8ee as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/526833cf71c141eaa83b713c8df6d8ee 2023-04-07 01:56:01,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,106 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,106 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,108 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,108 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,108 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 526833cf71c141eaa83b713c8df6d8ee(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,109 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,109 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832561078; duration=0sec 2023-04-07 01:56:01,109 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,109 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,109 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,116 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,116 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,116 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,128 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/8a9d5b5f0b0e4be2912374e0b09a0252 2023-04-07 01:56:01,149 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/35ab8a06eccc4b5e89f1053ea3e40bc8 2023-04-07 01:56:01,177 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/53d94e4232544593abf9db246925b29c 2023-04-07 01:56:01,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/8a9d5b5f0b0e4be2912374e0b09a0252 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252 2023-04-07 01:56:01,189 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 01:56:01,191 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/35ab8a06eccc4b5e89f1053ea3e40bc8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8 2023-04-07 01:56:01,199 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 01:56:01,200 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/53d94e4232544593abf9db246925b29c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c 2023-04-07 01:56:01,228 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 01:56:01,228 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 112ms, sequenceid=287, compaction requested=true 2023-04-07 01:56:01,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,228 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,232 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749979ed[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,236 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,237 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,241 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,241 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,241 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,243 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,243 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,247 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,247 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,247 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,247 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,247 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,248 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,248 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,248 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:01,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:01,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/49bb811442924b7e9376c5ab71a88ac6 2023-04-07 01:56:01,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/78d156cfbb9b4e6fab20b5279b976d87 2023-04-07 01:56:01,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/31d45437babf4720a47171ec289ed069 2023-04-07 01:56:01,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/49bb811442924b7e9376c5ab71a88ac6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6 2023-04-07 01:56:01,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6, entries=250, sequenceid=297, filesize=16.7 K 2023-04-07 01:56:01,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/78d156cfbb9b4e6fab20b5279b976d87 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87 2023-04-07 01:56:01,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87, entries=150, sequenceid=297, filesize=11.9 K 2023-04-07 01:56:01,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/31d45437babf4720a47171ec289ed069 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069 2023-04-07 01:56:01,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069, entries=150, sequenceid=297, filesize=11.9 K 2023-04-07 01:56:01,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561293, max timestamp=1680832561338Num uniques 859; 2023-04-07 01:56:01,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561293, max timestamp=1680832561338Num uniques 859; 2023-04-07 01:56:01,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=58.14 KB/59540 for 0e0b9000c3ed0a157bd0554455c64d06 in 64ms, sequenceid=297, compaction requested=true 2023-04-07 01:56:01,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,340 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:01,340 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:01,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561293, max timestamp=1680832561338Num uniques 859; 2023-04-07 01:56:01,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:01,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=126 KB 2023-04-07 01:56:01,341 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:01,342 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,342 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:01,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,342 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/60b35d9f1dfb4f369192e2a3119781d2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=55.1 K 2023-04-07 01:56:01,342 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:01,342 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:01,342 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 60b35d9f1dfb4f369192e2a3119781d2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680832560845 2023-04-07 01:56:01,343 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/16f16f8060494a67b469d145e175ad87, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.0 K 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f00903e1f3e14044afc366f7c20e7d01, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680832560902 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 16f16f8060494a67b469d145e175ad87, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680832560845 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8a9d5b5f0b0e4be2912374e0b09a0252, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680832560952 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 49bb811442924b7e9376c5ab71a88ac6, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561271 2023-04-07 01:56:01,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2c727ac98e3a4b2ea3872e958d582a63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680832560902 2023-04-07 01:56:01,344 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 35ab8a06eccc4b5e89f1053ea3e40bc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680832560952 2023-04-07 01:56:01,344 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 78d156cfbb9b4e6fab20b5279b976d87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561272 2023-04-07 01:56:01,348 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,352 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,352 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,356 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,356 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,356 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,369 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,372 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,372 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,372 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:01,372 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2b1c38[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,385 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,387 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,387 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,387 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,389 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/7ebf2b7dc18e467281b09907e818b3e3 2023-04-07 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,398 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/2ccd842f7976459abfe37d5d68fd5b7a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ccd842f7976459abfe37d5d68fd5b7a 2023-04-07 01:56:01,398 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,398 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561379, max timestamp=1680832561396Num uniques 859; 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561379, max timestamp=1680832561396Num uniques 859; 2023-04-07 01:56:01,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,399 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832561379, max timestamp=1680832561396Num uniques 859; 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,403 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/9f57caf3b77e4d29aa086c7df646b79d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/9f57caf3b77e4d29aa086c7df646b79d 2023-04-07 01:56:01,404 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 2ccd842f7976459abfe37d5d68fd5b7a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,404 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,404 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=12, startTime=1680832561340; duration=0sec 2023-04-07 01:56:01,404 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,404 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:01,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7d3cf132ac3d41c18f5c8bbd7006904c 2023-04-07 01:56:01,406 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:01,406 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:01,406 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,406 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/526833cf71c141eaa83b713c8df6d8ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=36.2 K 2023-04-07 01:56:01,407 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 526833cf71c141eaa83b713c8df6d8ee, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680832560902 2023-04-07 01:56:01,407 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 53d94e4232544593abf9db246925b29c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680832560952 2023-04-07 01:56:01,408 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 31d45437babf4720a47171ec289ed069, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561272 2023-04-07 01:56:01,408 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 9f57caf3b77e4d29aa086c7df646b79d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,408 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,409 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=12, startTime=1680832561340; duration=0sec 2023-04-07 01:56:01,409 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,417 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3d59fdcb07aa43f0824f79fe2144f7a3 2023-04-07 01:56:01,440 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7 2023-04-07 01:56:01,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/7ebf2b7dc18e467281b09907e818b3e3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832561423, max timestamp=1680832561440Num uniques 1018; 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832561423, max timestamp=1680832561440Num uniques 1018; 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680832561423, max timestamp=1680832561440Num uniques 1018; 2023-04-07 01:56:01,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-07 01:56:01,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3, entries=350, sequenceid=321, filesize=21.4 K 2023-04-07 01:56:01,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7d3cf132ac3d41c18f5c8bbd7006904c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c 2023-04-07 01:56:01,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c, entries=150, sequenceid=321, filesize=11.9 K 2023-04-07 01:56:01,453 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into ba6fe777d0ba421d9ad7a4d0d1cab4b7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,454 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,454 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832561341; duration=0sec 2023-04-07 01:56:01,454 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3d59fdcb07aa43f0824f79fe2144f7a3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3 2023-04-07 01:56:01,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3, entries=150, sequenceid=321, filesize=11.9 K 2023-04-07 01:56:01,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~125.95 KB/128976, currentSize=67.09 KB/68700 for 0e0b9000c3ed0a157bd0554455c64d06 in 125ms, sequenceid=321, compaction requested=false 2023-04-07 01:56:01,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:01,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=80.51 KB heapSize=220.08 KB 2023-04-07 01:56:01,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:01,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:01,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:01,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:01,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/eb064f1e1d6d4ab4bd0c248c0177b895 2023-04-07 01:56:01,498 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832621489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832621489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,504 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832621498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,511 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832621499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832621499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,514 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/aa4dd55268ff45ca8243a0261f22638f 2023-04-07 01:56:01,516 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,516 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,516 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:01,516 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,521 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724895aa[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,528 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,532 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,534 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,534 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,537 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,543 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,544 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,544 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/2b5f733b237a4f32aa9c35a378ddb59a 2023-04-07 01:56:01,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/eb064f1e1d6d4ab4bd0c248c0177b895 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895 2023-04-07 01:56:01,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895, entries=200, sequenceid=370, filesize=14.3 K 2023-04-07 01:56:01,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/aa4dd55268ff45ca8243a0261f22638f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f 2023-04-07 01:56:01,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f, entries=150, sequenceid=370, filesize=11.9 K 2023-04-07 01:56:01,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/2b5f733b237a4f32aa9c35a378ddb59a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a 2023-04-07 01:56:01,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a, entries=150, sequenceid=370, filesize=11.9 K 2023-04-07 01:56:01,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~239.95 KB/245712, currentSize=93.93 KB/96180 for 0e0b9000c3ed0a157bd0554455c64d06 in 98ms, sequenceid=370, compaction requested=true 2023-04-07 01:56:01,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,565 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:01,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,565 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:01,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,566 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:01,566 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:01,566 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:01,566 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:01,566 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,566 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,566 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ccd842f7976459abfe37d5d68fd5b7a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.3 K 2023-04-07 01:56:01,566 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/9f57caf3b77e4d29aa086c7df646b79d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=36.3 K 2023-04-07 01:56:01,566 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2ccd842f7976459abfe37d5d68fd5b7a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561272 2023-04-07 01:56:01,567 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9f57caf3b77e4d29aa086c7df646b79d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561272 2023-04-07 01:56:01,567 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7ebf2b7dc18e467281b09907e818b3e3, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832561293 2023-04-07 01:56:01,567 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7d3cf132ac3d41c18f5c8bbd7006904c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832561293 2023-04-07 01:56:01,567 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting eb064f1e1d6d4ab4bd0c248c0177b895, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561450 2023-04-07 01:56:01,567 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting aa4dd55268ff45ca8243a0261f22638f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561464 2023-04-07 01:56:01,573 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,575 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,591 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d7575f9555fa4dc2927da8cc6aa73e7c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d7575f9555fa4dc2927da8cc6aa73e7c 2023-04-07 01:56:01,595 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into d7575f9555fa4dc2927da8cc6aa73e7c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,595 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,595 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832561565; duration=0sec 2023-04-07 01:56:01,595 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:01,595 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:01,596 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:01,596 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:01,596 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,596 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=36.3 K 2023-04-07 01:56:01,596 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ba6fe777d0ba421d9ad7a4d0d1cab4b7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680832561272 2023-04-07 01:56:01,597 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3d59fdcb07aa43f0824f79fe2144f7a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680832561293 2023-04-07 01:56:01,597 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2b5f733b237a4f32aa9c35a378ddb59a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561464 2023-04-07 01:56:01,602 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:01,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:01,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9c008554c0214fd6af08db1476ec3293 2023-04-07 01:56:01,623 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/1e077ac84afb46a8b2673fcc49f33970 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e077ac84afb46a8b2673fcc49f33970 2023-04-07 01:56:01,628 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 1e077ac84afb46a8b2673fcc49f33970(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:01,628 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:01,628 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832561565; duration=0sec 2023-04-07 01:56:01,628 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:01,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832621623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37302 deadline: 1680832621625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832621631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832621632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832621632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,644 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,649 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,650 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,651 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:01,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8b0696[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,661 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,662 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,665 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,665 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,665 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,666 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,666 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,666 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,679 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,679 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832621733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832621733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832621741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832621741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37372 deadline: 1680832621743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,779 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,788 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,792 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,792 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,793 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:01,795 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,797 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e24a1b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,801 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,804 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,810 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,810 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,816 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:01,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:01,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,827 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,827 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:01,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:01,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832621946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37300 deadline: 1680832621946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832621946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,956 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:01,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:01,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:01,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832621957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:01,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832621957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:01,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:01,976 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,977 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,978 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:01,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:01,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:01,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:01,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:01,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:01,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:01,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:01,979 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:01,980 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,980 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:01,980 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:01,980 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:01,986 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:01,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733f5063[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:01,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:01,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,992 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,992 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:01,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:01,994 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:01,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:01,997 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:01,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:01,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:01,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:01,997 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:01,998 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b0ab678475cf4c5c99f3323f23462ae5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b0ab678475cf4c5c99f3323f23462ae5 2023-04-07 01:56:01,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:01,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:02,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,002 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,003 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into b0ab678475cf4c5c99f3323f23462ae5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:02,003 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:02,003 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832561565; duration=0sec 2023-04-07 01:56:02,004 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:02,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:02,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:02,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:02,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:02,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:02,006 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:02,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:02,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,006 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:02,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:02,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:02,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:02,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:02,007 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:02,007 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:02,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/1b340ea7202548599905d1f4c17d7898 2023-04-07 01:56:02,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/d41029154e4b45fc80afe2b63ca960fc 2023-04-07 01:56:02,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9c008554c0214fd6af08db1476ec3293 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293 2023-04-07 01:56:02,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293, entries=200, sequenceid=389, filesize=14.3 K 2023-04-07 01:56:02,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/1b340ea7202548599905d1f4c17d7898 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898 2023-04-07 01:56:02,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898, entries=150, sequenceid=389, filesize=11.9 K 2023-04-07 01:56:02,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/d41029154e4b45fc80afe2b63ca960fc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc 2023-04-07 01:56:02,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc, entries=150, sequenceid=389, filesize=11.9 K 2023-04-07 01:56:02,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 0e0b9000c3ed0a157bd0554455c64d06 in 499ms, sequenceid=389, compaction requested=false 2023-04-07 01:56:02,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:02,107 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:02,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:02,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:02,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:02,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:02,122 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:02,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:02,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:02,123 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:02,123 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:02,123 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:02,124 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:02,124 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:02,124 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:02,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:02,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:02,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:02,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:02,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:02,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:02,136 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 01:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:02,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:02,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:02,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:02,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:02,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:02,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832622275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832622276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832622280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37358 deadline: 1680832622281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832622284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832622386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37336 deadline: 1680832622387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832622387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832622387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832622387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,547 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/0b8f2ccdff744ac8b1e7d036ff2aefb8 2023-04-07 01:56:02,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832622592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832622593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832622593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832622593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832622594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832622894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832622904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832622905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832622905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:02,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832622905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:02,972 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0b86e70274e9466ebb27b1c7fb8da0b0 2023-04-07 01:56:03,387 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/5d4b37254dfa4c9da3b3df221183986d 2023-04-07 01:56:03,391 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/0b8f2ccdff744ac8b1e7d036ff2aefb8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8 2023-04-07 01:56:03,395 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8, entries=150, sequenceid=406, filesize=11.9 K 2023-04-07 01:56:03,396 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/0b86e70274e9466ebb27b1c7fb8da0b0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0 2023-04-07 01:56:03,409 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0, entries=150, sequenceid=406, filesize=11.9 K 2023-04-07 01:56:03,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832623404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,411 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/5d4b37254dfa4c9da3b3df221183986d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d 2023-04-07 01:56:03,415 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d, entries=150, sequenceid=406, filesize=11.9 K 2023-04-07 01:56:03,416 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 0e0b9000c3ed0a157bd0554455c64d06 in 1280ms, sequenceid=406, compaction requested=true 2023-04-07 01:56:03,416 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:03,416 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:03,419 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:03,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:03,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456c7e6b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:03,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:03,422 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:03,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:03,424 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:03,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:03,430 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:03,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:03,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:03,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:03,431 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:03,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/06eba8ca9190466aac3bbdf0036e8c78 2023-04-07 01:56:03,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,434 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:03,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:03,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,439 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:03,439 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832623437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832623438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832623444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/befb389f756b438095f6d22cb946a105 2023-04-07 01:56:03,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832623445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,539 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:03,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:03,542 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:03,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,543 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,545 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,545 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:03,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:03,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:03,546 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,546 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:03,546 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:03,546 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:03,549 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,549 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,550 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,550 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832623546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d06fe6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:03,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:03,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832623548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832623549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,553 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:03,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:03,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:03,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:03,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:03,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:03,555 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:03,559 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:03,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:03,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:03,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:03,559 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:03,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832623554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,561 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,561 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,562 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,569 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,569 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,569 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,570 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:03,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:03,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,570 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:03,570 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:03,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:03,670 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:03,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:03,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:03,674 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:03,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:03,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:03,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,675 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:03,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:03,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:03,678 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:03,679 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301457b5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:03,681 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:03,681 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:03,693 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:03,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:03,693 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:03,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,695 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:03,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:03,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:03,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,701 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:03,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,701 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:03,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832623753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832623753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832623753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832623761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,801 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:03,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:03,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:03,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:03,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:03,805 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:03,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:03,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:03,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,808 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:03,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:03,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:03,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:03,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:03,809 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:03,809 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,809 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:03,810 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:03,810 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:03,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:03,812 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3921b59c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:03,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:03,814 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,815 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:03,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:03,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:03,817 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:03,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:03,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:03,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:03,817 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:03,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:03,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:03,820 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:03,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:03,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:03,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:03,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:03,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:03,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,825 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:03,825 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:03,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3330c376867247328d06c044c50fd1ac 2023-04-07 01:56:03,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/06eba8ca9190466aac3bbdf0036e8c78 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78 2023-04-07 01:56:03,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78, entries=200, sequenceid=424, filesize=14.3 K 2023-04-07 01:56:03,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/befb389f756b438095f6d22cb946a105 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105 2023-04-07 01:56:03,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105, entries=150, sequenceid=424, filesize=11.9 K 2023-04-07 01:56:03,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/3330c376867247328d06c044c50fd1ac as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac 2023-04-07 01:56:03,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac, entries=150, sequenceid=424, filesize=11.9 K 2023-04-07 01:56:03,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0e0b9000c3ed0a157bd0554455c64d06 in 486ms, sequenceid=424, compaction requested=true 2023-04-07 01:56:03,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:03,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:03,906 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:03,906 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:03,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:03,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:03,907 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:03,908 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:03,908 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,908 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d7575f9555fa4dc2927da8cc6aa73e7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.3 K 2023-04-07 01:56:03,908 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:03,908 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d7575f9555fa4dc2927da8cc6aa73e7c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561464 2023-04-07 01:56:03,908 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:03,908 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,908 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b0ab678475cf4c5c99f3323f23462ae5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=53.1 K 2023-04-07 01:56:03,909 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 1b340ea7202548599905d1f4c17d7898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680832561488 2023-04-07 01:56:03,909 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b0ab678475cf4c5c99f3323f23462ae5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561464 2023-04-07 01:56:03,909 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0b86e70274e9466ebb27b1c7fb8da0b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680832561623 2023-04-07 01:56:03,909 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting befb389f756b438095f6d22cb946a105, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:03,909 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9c008554c0214fd6af08db1476ec3293, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680832561488 2023-04-07 01:56:03,910 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0b8f2ccdff744ac8b1e7d036ff2aefb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680832561623 2023-04-07 01:56:03,911 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 06eba8ca9190466aac3bbdf0036e8c78, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:03,919 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:03,925 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:03,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:03,926 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#495 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:03,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:03,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:03,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:03,933 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:03,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:03,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:03,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:03,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:03,935 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:03,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:03,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:03,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:03,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:03,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:03,937 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:03,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:03,941 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:03,941 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,941 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:03,943 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:03,943 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:03,958 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9a4b046f2ea9478e848783616192de94 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9a4b046f2ea9478e848783616192de94 2023-04-07 01:56:03,963 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 9a4b046f2ea9478e848783616192de94(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:03,963 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:03,963 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=12, startTime=1680832563906; duration=0sec 2023-04-07 01:56:03,963 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:03,963 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:03,969 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:03,969 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:03,969 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:03,969 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e077ac84afb46a8b2673fcc49f33970, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.3 K 2023-04-07 01:56:03,969 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1e077ac84afb46a8b2673fcc49f33970, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680832561464 2023-04-07 01:56:03,969 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d41029154e4b45fc80afe2b63ca960fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680832561488 2023-04-07 01:56:03,970 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5d4b37254dfa4c9da3b3df221183986d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680832561623 2023-04-07 01:56:03,970 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3330c376867247328d06c044c50fd1ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:03,970 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/814558d09c4f471f86324cc73ad3ddec 2023-04-07 01:56:03,976 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:03,989 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/94a1c555c3364e94bddd35ee8f905889 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/94a1c555c3364e94bddd35ee8f905889 2023-04-07 01:56:03,994 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 94a1c555c3364e94bddd35ee8f905889(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:03,994 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:03,994 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=12, startTime=1680832563906; duration=0sec 2023-04-07 01:56:03,994 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:03,997 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/3c0efc82d4b440d392fdad0cc8532a33 2023-04-07 01:56:04,014 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/6e929d3dba8f44ada1b2e0d6572f676a 2023-04-07 01:56:04,017 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/814558d09c4f471f86324cc73ad3ddec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec 2023-04-07 01:56:04,020 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec, entries=150, sequenceid=438, filesize=11.9 K 2023-04-07 01:56:04,021 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/3c0efc82d4b440d392fdad0cc8532a33 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33 2023-04-07 01:56:04,025 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33, entries=150, sequenceid=438, filesize=11.9 K 2023-04-07 01:56:04,026 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/6e929d3dba8f44ada1b2e0d6572f676a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a 2023-04-07 01:56:04,031 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a, entries=150, sequenceid=438, filesize=11.9 K 2023-04-07 01:56:04,032 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 89ms, sequenceid=438, compaction requested=false 2023-04-07 01:56:04,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:04,032 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,036 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f180bf5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,041 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,041 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,042 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,043 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,047 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,047 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,047 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,048 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,054 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,054 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,055 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,055 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:04,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:04,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9af191be1f5c40838538d4c895925811 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832564088, max timestamp=1680832564101Num uniques 859; 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832564088, max timestamp=1680832564101Num uniques 859; 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832564088, max timestamp=1680832564101Num uniques 859; 2023-04-07 01:56:04,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:04,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/65f5f73a62e6435abd1b50092e74eeda 2023-04-07 01:56:04,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/667cd7d13add47499910db2d606ee216 2023-04-07 01:56:04,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9af191be1f5c40838538d4c895925811 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811 2023-04-07 01:56:04,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811, entries=200, sequenceid=450, filesize=14.3 K 2023-04-07 01:56:04,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/65f5f73a62e6435abd1b50092e74eeda as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda 2023-04-07 01:56:04,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda, entries=150, sequenceid=450, filesize=11.9 K 2023-04-07 01:56:04,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/667cd7d13add47499910db2d606ee216 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216 2023-04-07 01:56:04,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216, entries=150, sequenceid=450, filesize=11.9 K 2023-04-07 01:56:04,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=127.47 KB/130530 for 0e0b9000c3ed0a157bd0554455c64d06 in 78ms, sequenceid=450, compaction requested=true 2023-04-07 01:56:04,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:04,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:04,146 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:04,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:04,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 01:56:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:04,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:04,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:04,147 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:04,147 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:04,147 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,147 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9a4b046f2ea9478e848783616192de94, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=38.9 K 2023-04-07 01:56:04,147 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9a4b046f2ea9478e848783616192de94, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:04,148 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 814558d09c4f471f86324cc73ad3ddec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1680832563436 2023-04-07 01:56:04,148 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9af191be1f5c40838538d4c895925811, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680832564062 2023-04-07 01:56:04,155 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,180 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,181 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,181 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:04,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,183 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,184 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832624180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832624181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832624183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832624184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,194 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b96c4bf5d2f24ff9955a8850693899a3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b96c4bf5d2f24ff9955a8850693899a3 2023-04-07 01:56:04,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,198 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into b96c4bf5d2f24ff9955a8850693899a3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:04,198 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:04,198 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832564146; duration=0sec 2023-04-07 01:56:04,198 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 01:56:04,198 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:04,199 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:04,199 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:04,199 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:04,199 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:04,200 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:04,200 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:04,200 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,200 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/94a1c555c3364e94bddd35ee8f905889, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=36.6 K 2023-04-07 01:56:04,200 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 94a1c555c3364e94bddd35ee8f905889, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,201 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,201 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,201 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,201 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,201 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6e929d3dba8f44ada1b2e0d6572f676a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1680832563436 2023-04-07 01:56:04,201 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 667cd7d13add47499910db2d606ee216, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680832564062 2023-04-07 01:56:04,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,202 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,204 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,205 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af2e8af[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,207 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,207 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,208 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:04,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,218 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,218 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,218 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,219 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,219 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,219 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,220 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:04,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,224 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,224 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,224 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832624292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832624292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832624294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832624295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,324 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,327 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,331 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,331 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,331 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,331 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,334 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,334 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4849a5bb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,339 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,341 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,341 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,347 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,348 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,348 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,348 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,348 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,349 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/5497a4eaf6a44542942e4b0cd67dac16 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5497a4eaf6a44542942e4b0cd67dac16 2023-04-07 01:56:04,353 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 5497a4eaf6a44542942e4b0cd67dac16(size=12.8 K), total size for store is 36.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:04,353 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:04,353 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=12, startTime=1680832563906; duration=0sec 2023-04-07 01:56:04,353 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:04,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832624414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,449 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,451 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,452 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,453 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,453 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,453 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,453 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,455 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,455 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,455 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,455 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,455 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,457 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,457 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,458 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,458 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf0647b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,460 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,462 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,464 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,464 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,464 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,465 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,465 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,465 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,467 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,469 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,469 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:04,470 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,470 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,470 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832624501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832624502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832624502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832624503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,570 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,573 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,573 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,575 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,575 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,576 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,577 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,577 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,577 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,577 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,580 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e89fb3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,582 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,582 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,583 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,583 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,585 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,587 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,587 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,587 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,589 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,589 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,589 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,589 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,590 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,595 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:04,595 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,596 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,596 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,596 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,596 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,596 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c0f004a916814fc39525473b08903fe8 2023-04-07 01:56:04,621 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7a70db1fc9e44df2b833b43cbe89686d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a70db1fc9e44df2b833b43cbe89686d 2023-04-07 01:56:04,625 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 7a70db1fc9e44df2b833b43cbe89686d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:04,625 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:04,625 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832564146; duration=0sec 2023-04-07 01:56:04,625 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:04,696 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,699 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,701 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,703 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,705 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e12acf1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,708 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,708 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,709 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,709 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,713 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,713 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,713 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,713 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,717 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,722 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,722 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,722 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832624806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832624806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832624807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:04,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832624812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,822 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,826 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,828 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,828 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,829 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,829 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,829 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,829 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,832 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,833 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf3a8f5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,833 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,835 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,836 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,839 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,839 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,840 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,840 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,840 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,841 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,847 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,848 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,848 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,848 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,848 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,848 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:04,948 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:04,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:04,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:04,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:04,953 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:04,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:04,953 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:04,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,955 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,956 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:04,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:04,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:04,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:04,956 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:04,957 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,957 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:04,957 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:04,957 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:04,959 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145fca4c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:04,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:04,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,962 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:04,962 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,962 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:04,967 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:04,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:04,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:04,967 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:04,967 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:04,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,968 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,969 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,970 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:04,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:04,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:04,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:04,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:04,976 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:04,976 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:04,977 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:04,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:04,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:05,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7a9df67f281f474ea716f1a7409d3c96 2023-04-07 01:56:05,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/28f7f485b374463b92373008640b39b1 2023-04-07 01:56:05,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c0f004a916814fc39525473b08903fe8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8 2023-04-07 01:56:05,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8, entries=300, sequenceid=487, filesize=19.0 K 2023-04-07 01:56:05,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/7a9df67f281f474ea716f1a7409d3c96 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96 2023-04-07 01:56:05,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96, entries=150, sequenceid=487, filesize=11.9 K 2023-04-07 01:56:05,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/28f7f485b374463b92373008640b39b1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1 2023-04-07 01:56:05,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1, entries=150, sequenceid=487, filesize=11.9 K 2023-04-07 01:56:05,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 0e0b9000c3ed0a157bd0554455c64d06 in 903ms, sequenceid=487, compaction requested=true 2023-04-07 01:56:05,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:05,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:05,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:05,050 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. because compaction request was cancelled 2023-04-07 01:56:05,052 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,052 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5497a4eaf6a44542942e4b0cd67dac16, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.5 K 2023-04-07 01:56:05,052 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5497a4eaf6a44542942e4b0cd67dac16, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680832562272 2023-04-07 01:56:05,053 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3c0efc82d4b440d392fdad0cc8532a33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1680832563436 2023-04-07 01:56:05,053 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 65f5f73a62e6435abd1b50092e74eeda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680832564062 2023-04-07 01:56:05,053 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7a9df67f281f474ea716f1a7409d3c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680832564143 2023-04-07 01:56:05,059 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:05,068 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/e33fcecd4e6c428f9558a659d75a12a2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e33fcecd4e6c428f9558a659d75a12a2 2023-04-07 01:56:05,073 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into e33fcecd4e6c428f9558a659d75a12a2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:05,073 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,073 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=12, startTime=1680832565050; duration=0sec 2023-04-07 01:56:05,073 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:05,077 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,080 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,083 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,085 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,085 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:05,086 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,094 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9db45593624c449fbc276fe3be8ce586 2023-04-07 01:56:05,104 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d6efe7e6793e499b88617b6fe30fe927 2023-04-07 01:56:05,140 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/a7d23a64e619417bbea9bf0136c9fbee 2023-04-07 01:56:05,144 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/9db45593624c449fbc276fe3be8ce586 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586 2023-04-07 01:56:05,152 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586, entries=150, sequenceid=500, filesize=11.9 K 2023-04-07 01:56:05,153 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d6efe7e6793e499b88617b6fe30fe927 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927 2023-04-07 01:56:05,167 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927, entries=150, sequenceid=500, filesize=11.9 K 2023-04-07 01:56:05,168 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/a7d23a64e619417bbea9bf0136c9fbee as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee 2023-04-07 01:56:05,177 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee, entries=150, sequenceid=500, filesize=11.9 K 2023-04-07 01:56:05,178 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 93ms, sequenceid=500, compaction requested=true 2023-04-07 01:56:05,178 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,178 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:05,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73adda73[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:05,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:05,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,184 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:05,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,184 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:05,186 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:05,189 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:05,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:05,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:05,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:05,189 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:05,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,196 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,197 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,197 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,197 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:05,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:05,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,197 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:05,197 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:05,297 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,301 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,301 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,303 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,303 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:05,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:05,329 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:05,329 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:05,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a395d3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:05,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:05,334 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,337 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:05,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:05,337 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:05,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:05,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:05,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,338 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,338 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:05,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:05,341 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:05,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:05,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:05,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:05,341 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:05,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,347 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,349 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,351 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,351 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:05,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:05,352 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:05,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,352 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:05,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b9fbd69894c84aba852dd19d2906b4a3 2023-04-07 01:56:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680832565355, max timestamp=1680832565368Num uniques 859; 2023-04-07 01:56:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:05,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680832565355, max timestamp=1680832565368Num uniques 859; 2023-04-07 01:56:05,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:05,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680832565355, max timestamp=1680832565368Num uniques 859; 2023-04-07 01:56:05,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:05,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832625392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832625393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832625395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832625396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,452 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,459 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,459 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,459 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,459 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,460 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,462 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,462 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,463 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,485 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,486 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,486 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:05,486 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:05,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:05,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:05,486 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:05,488 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,489 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:05,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee13032[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:05,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:05,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,495 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:05,495 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,495 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:05,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:05,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:05,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:05,499 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:05,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:05,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:05,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:05,499 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:05,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:05,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,506 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,506 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:05,507 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:05,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:05,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832625502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832625503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832625503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832625504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,607 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,612 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,636 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,636 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:05,646 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,646 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,647 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,647 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14069808[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:05,647 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:05,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:05,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,651 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:05,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:05,653 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:05,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:05,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:05,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:05,653 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:05,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,663 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,663 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,663 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,663 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,663 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:05,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:05,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,664 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:05,664 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:05,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832625711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832625712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832625712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:05,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832625712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,764 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,767 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,767 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,768 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,769 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,770 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:05,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,773 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:05,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e5b6b9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:05,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,775 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:05,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:05,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:05,775 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,775 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:05,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/ae2aa76924e84653abe3d8a6795baef6 2023-04-07 01:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:05,782 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:05,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:05,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:05,782 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:05,782 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:05,783 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,783 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,784 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:05,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:05,785 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/9425e7f05bfa4a7db4d749b19358ef7b 2023-04-07 01:56:05,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,790 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:05,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:05,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:05,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:05,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:05,791 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:05,791 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:05,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,792 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:05,792 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b9fbd69894c84aba852dd19d2906b4a3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3 2023-04-07 01:56:05,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3, entries=200, sequenceid=511, filesize=14.3 K 2023-04-07 01:56:05,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/ae2aa76924e84653abe3d8a6795baef6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6 2023-04-07 01:56:05,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6, entries=100, sequenceid=511, filesize=9.5 K 2023-04-07 01:56:05,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/9425e7f05bfa4a7db4d749b19358ef7b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b 2023-04-07 01:56:05,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b, entries=100, sequenceid=511, filesize=9.5 K 2023-04-07 01:56:05,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 0e0b9000c3ed0a157bd0554455c64d06 in 478ms, sequenceid=511, compaction requested=true 2023-04-07 01:56:05,804 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:05,804 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:05,804 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:05,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:05,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:05,806 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:05,806 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:05,806 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,806 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e33fcecd4e6c428f9558a659d75a12a2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.3 K 2023-04-07 01:56:05,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59488 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:05,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:05,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,807 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b96c4bf5d2f24ff9955a8850693899a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=58.1 K 2023-04-07 01:56:05,807 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b96c4bf5d2f24ff9955a8850693899a3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680832564062 2023-04-07 01:56:05,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e33fcecd4e6c428f9558a659d75a12a2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680832564143 2023-04-07 01:56:05,808 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c0f004a916814fc39525473b08903fe8, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680832564119 2023-04-07 01:56:05,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d6efe7e6793e499b88617b6fe30fe927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680832564160 2023-04-07 01:56:05,808 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9db45593624c449fbc276fe3be8ce586, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680832564160 2023-04-07 01:56:05,809 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ae2aa76924e84653abe3d8a6795baef6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832565323 2023-04-07 01:56:05,809 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b9fbd69894c84aba852dd19d2906b4a3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832565319 2023-04-07 01:56:05,815 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#515 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:05,818 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#516 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:05,828 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/d9275ec5214f40e4a4d29e3f9b0d4d48 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d9275ec5214f40e4a4d29e3f9b0d4d48 2023-04-07 01:56:05,832 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into d9275ec5214f40e4a4d29e3f9b0d4d48(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:05,832 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,832 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=12, startTime=1680832565804; duration=0sec 2023-04-07 01:56:05,832 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:05,832 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:05,834 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:05,834 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:05,835 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,835 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a70db1fc9e44df2b833b43cbe89686d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=46.2 K 2023-04-07 01:56:05,835 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7a70db1fc9e44df2b833b43cbe89686d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680832564062 2023-04-07 01:56:05,835 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 28f7f485b374463b92373008640b39b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680832564143 2023-04-07 01:56:05,836 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting a7d23a64e619417bbea9bf0136c9fbee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680832564160 2023-04-07 01:56:05,836 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9425e7f05bfa4a7db4d749b19358ef7b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832565323 2023-04-07 01:56:05,845 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:05,855 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/c0b28de473e34fda9df700de4bd9c07a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/c0b28de473e34fda9df700de4bd9c07a 2023-04-07 01:56:05,859 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into c0b28de473e34fda9df700de4bd9c07a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:05,859 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:05,859 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=12, startTime=1680832565804; duration=0sec 2023-04-07 01:56:05,859 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:05,891 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:05,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:05,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:05,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:05,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:05,895 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:05,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:05,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:05,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:05,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:05,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:05,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:05,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:05,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:05,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:05,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:05,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:05,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:05,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:05,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:05,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:05,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:05,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:05,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:05,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:05,900 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:05,900 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:06,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:06,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:06,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832626038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832626038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832626039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832626039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832626140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832626140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832626152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832626152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,230 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/a6aad2871caa4df389bb2528effe7cd3 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a6aad2871caa4df389bb2528effe7cd3 2023-04-07 01:56:06,234 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into a6aad2871caa4df389bb2528effe7cd3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:06,234 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:06,234 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832565804; duration=0sec 2023-04-07 01:56:06,234 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:06,314 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/dc8ea3404c3b4ceaa1a47da1245b2f03 2023-04-07 01:56:06,325 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/00d0ca18cac24fd8a7b52a7809c57194 2023-04-07 01:56:06,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832626345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832626346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832626357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832626359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832626424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,429 DEBUG [Thread-3165] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:06,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832626655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832626656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832626664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:06,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832626666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,751 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/83e5f2bd42a44c3399425c8d04cf10e5 2023-04-07 01:56:06,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/dc8ea3404c3b4ceaa1a47da1245b2f03 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03 2023-04-07 01:56:06,761 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 01:56:06,762 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/00d0ca18cac24fd8a7b52a7809c57194 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194 2023-04-07 01:56:06,767 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 01:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/83e5f2bd42a44c3399425c8d04cf10e5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5 2023-04-07 01:56:06,774 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 01:56:06,775 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=60.38 KB/61830 for 0e0b9000c3ed0a157bd0554455c64d06 in 875ms, sequenceid=547, compaction requested=false 2023-04-07 01:56:06,775 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:06,775 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:06,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:06,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:06,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:06,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:06,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d7cb75b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:06,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:06,789 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:06,792 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,792 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:06,793 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:06,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:06,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:06,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:06,796 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:06,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:06,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:06,796 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:06,796 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:06,798 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,798 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:06,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,799 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:06,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:06,805 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:06,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:06,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:06,805 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:06,805 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:06,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:06,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:06,905 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:06,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:06,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:06,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:06,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:06,908 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:06,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:06,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:06,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:06,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:06,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:06,910 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:06,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:06,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:06,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:06,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:06,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:06,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:06,913 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:06,914 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:06,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:06,916 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:06,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:06,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:06,916 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 01:56:06,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:06,916 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:06,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:06,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:06,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:06,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:06,926 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b37220456711490bac2fc46d90b5fb28 2023-04-07 01:56:07,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:07,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832627203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832627204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832627205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832627209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832627310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832627314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832627314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832627322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,353 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/815a1df574ea4f0697be157bb4234ef6 2023-04-07 01:56:07,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f51133eb3196455f847a9cc48328a5d9 2023-04-07 01:56:07,400 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b37220456711490bac2fc46d90b5fb28 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28 2023-04-07 01:56:07,404 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28, entries=150, sequenceid=560, filesize=11.9 K 2023-04-07 01:56:07,405 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/815a1df574ea4f0697be157bb4234ef6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6 2023-04-07 01:56:07,409 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6, entries=150, sequenceid=560, filesize=11.9 K 2023-04-07 01:56:07,410 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f51133eb3196455f847a9cc48328a5d9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9 2023-04-07 01:56:07,417 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9, entries=150, sequenceid=560, filesize=11.9 K 2023-04-07 01:56:07,418 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0e0b9000c3ed0a157bd0554455c64d06 in 502ms, sequenceid=560, compaction requested=true 2023-04-07 01:56:07,418 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:07,418 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:07,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6b6466[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:07,427 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:07,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,432 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:07,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:07,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:07,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:07,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:07,438 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:07,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:07,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:07,438 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:07,438 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:07,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,445 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:07,445 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:07,445 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:07,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:07,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:07,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:07,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:07,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/e5a935e713dd4b75a2d693f332017ae5 2023-04-07 01:56:07,545 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:07,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:07,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:07,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832627540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832627540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832627540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:07,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:07,555 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:07,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:07,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:07,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832627551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,558 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:07,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:07,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:07,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:07,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:07,564 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,564 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,565 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0d67fb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:07,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:07,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,568 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:07,568 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,569 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:07,570 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:07,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:07,574 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:07,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:07,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:07,574 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:07,574 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:07,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,577 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/04df03f02df7417e92a8610b50ecb5d9 2023-04-07 01:56:07,580 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:07,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,584 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:07,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,584 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:07,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:07,584 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832627652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832627653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832627654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832627657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,684 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:07,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:07,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:07,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:07,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:07,688 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:07,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:07,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:07,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,688 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,689 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:07,691 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:07,692 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,692 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,694 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e79407d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:07,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:07,695 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,697 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:07,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:07,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:07,699 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:07,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:07,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:07,700 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:07,700 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:07,703 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,703 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,703 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,703 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,704 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:07,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,707 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,707 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:07,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,707 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:07,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,807 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:07,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:07,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:07,816 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:07,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:07,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:07,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,818 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:07,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:07,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:07,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:07,840 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,840 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:07,840 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:07,840 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:07,842 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,842 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,843 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c7e486[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:07,844 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:07,845 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,846 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:07,846 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,846 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:07,851 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:07,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:07,854 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:07,854 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:07,854 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:07,854 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:07,854 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:07,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,860 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,861 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,862 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,864 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,864 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,864 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,864 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,865 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,865 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,865 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832627859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832627860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832627861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,864 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:07,865 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,866 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:07,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:07,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:07,866 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:07,866 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:07,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:07,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832627867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,966 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:07,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:07,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:07,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:07,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:07,980 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:07,980 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,980 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:07,980 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:07,980 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,980 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,982 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:07,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:07,982 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:07,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:07,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,984 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:07,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:07,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:07,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:07,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:07,988 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:07,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:07,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:07,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:07,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:07,992 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273efa6c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:07,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:07,997 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,999 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/2ec64f8389e34059be7a530c3a2ff416 2023-04-07 01:56:07,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:07,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:07,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:07,999 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:08,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,000 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:08,002 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:08,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/e5a935e713dd4b75a2d693f332017ae5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5 2023-04-07 01:56:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:08,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:08,005 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:08,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:08,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:08,005 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:08,005 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:08,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5, entries=200, sequenceid=581, filesize=14.3 K 2023-04-07 01:56:08,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:08,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/04df03f02df7417e92a8610b50ecb5d9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9 2023-04-07 01:56:08,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:08,007 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:08,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9, entries=150, sequenceid=581, filesize=11.9 K 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:08,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:08,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,013 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:08,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:08,013 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:08,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/2ec64f8389e34059be7a530c3a2ff416 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416 2023-04-07 01:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:08,013 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:08,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416, entries=150, sequenceid=581, filesize=11.9 K 2023-04-07 01:56:08,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 0e0b9000c3ed0a157bd0554455c64d06 in 510ms, sequenceid=581, compaction requested=true 2023-04-07 01:56:08,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:08,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:08,034 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:08,034 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:08,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:08,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52308 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:08,038 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:08,038 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d9275ec5214f40e4a4d29e3f9b0d4d48, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=51.1 K 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:08,038 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,038 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a6aad2871caa4df389bb2528effe7cd3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.7 K 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d9275ec5214f40e4a4d29e3f9b0d4d48, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832564183 2023-04-07 01:56:08,038 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a6aad2871caa4df389bb2528effe7cd3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832564183 2023-04-07 01:56:08,039 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting dc8ea3404c3b4ceaa1a47da1245b2f03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680832565391 2023-04-07 01:56:08,039 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 00d0ca18cac24fd8a7b52a7809c57194, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680832565391 2023-04-07 01:56:08,040 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b37220456711490bac2fc46d90b5fb28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680832566026 2023-04-07 01:56:08,040 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e5a935e713dd4b75a2d693f332017ae5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:08,040 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 815a1df574ea4f0697be157bb4234ef6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680832566026 2023-04-07 01:56:08,041 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 04df03f02df7417e92a8610b50ecb5d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:08,050 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:08,062 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#528 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:08,072 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/910a7fe7401f46e4a469b7fa6f900373 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/910a7fe7401f46e4a469b7fa6f900373 2023-04-07 01:56:08,076 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 910a7fe7401f46e4a469b7fa6f900373(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:08,076 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:08,076 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=12, startTime=1680832568034; duration=0sec 2023-04-07 01:56:08,076 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:08,076 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:08,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:08,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:08,077 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,077 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/c0b28de473e34fda9df700de4bd9c07a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=48.7 K 2023-04-07 01:56:08,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c0b28de473e34fda9df700de4bd9c07a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1680832564183 2023-04-07 01:56:08,078 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 83e5f2bd42a44c3399425c8d04cf10e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680832565391 2023-04-07 01:56:08,078 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f51133eb3196455f847a9cc48328a5d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680832566026 2023-04-07 01:56:08,079 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2ec64f8389e34059be7a530c3a2ff416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:08,102 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:08,117 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:08,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:08,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:08,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:08,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:08,122 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:08,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:08,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:08,122 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:08,124 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/bee7e0bde97d46f0b1c353f63bf905da as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/bee7e0bde97d46f0b1c353f63bf905da 2023-04-07 01:56:08,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,127 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:08,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,128 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/9ec89dd85c644bbca507e9fc106cea18 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9ec89dd85c644bbca507e9fc106cea18 2023-04-07 01:56:08,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,128 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:08,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:08,134 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 9ec89dd85c644bbca507e9fc106cea18(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:08,134 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:08,134 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=12, startTime=1680832568034; duration=0sec 2023-04-07 01:56:08,134 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:08,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:08,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:08,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:08,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:08,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:08,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:08,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:08,137 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:08,137 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into bee7e0bde97d46f0b1c353f63bf905da(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:08,137 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:08,137 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=12, startTime=1680832568034; duration=0sec 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:08,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:08,137 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:08,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:08,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:08,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680832568183, max timestamp=1680832568195Num uniques 859; 2023-04-07 01:56:08,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832568183, max timestamp=1680832568195Num uniques 859; 2023-04-07 01:56:08,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 01:56:08,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:08,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832568183, max timestamp=1680832568195Num uniques 859; 2023-04-07 01:56:08,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:08,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832628217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832628223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832628224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832628225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832628327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832628328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832628332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832628332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832628534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832628536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832628537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832628538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,564 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b57ccbe67d654891b5025ab62fcc327c 2023-04-07 01:56:08,574 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d1ed5be6ac154a20bc4871704d94eddf 2023-04-07 01:56:08,586 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/21364a73822f4600a1d386cc92c1886d 2023-04-07 01:56:08,589 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/b57ccbe67d654891b5025ab62fcc327c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c 2023-04-07 01:56:08,592 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c, entries=100, sequenceid=595, filesize=9.5 K 2023-04-07 01:56:08,593 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d1ed5be6ac154a20bc4871704d94eddf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf 2023-04-07 01:56:08,596 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf, entries=100, sequenceid=595, filesize=9.5 K 2023-04-07 01:56:08,596 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/21364a73822f4600a1d386cc92c1886d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d 2023-04-07 01:56:08,599 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d, entries=100, sequenceid=595, filesize=9.5 K 2023-04-07 01:56:08,600 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 0e0b9000c3ed0a157bd0554455c64d06 in 463ms, sequenceid=595, compaction requested=false 2023-04-07 01:56:08,600 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:08,600 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:08,602 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d63cf[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:08,604 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:08,604 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,606 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:08,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:08,609 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:08,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:08,612 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:08,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:08,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:08,612 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:08,612 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:08,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,614 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:08,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,615 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:08,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:08,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:08,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:08,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:08,619 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:08,619 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:08,619 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:08,719 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:08,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:08,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:08,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:08,723 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:08,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:08,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:08,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:08,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,725 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,725 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:08,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,726 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,726 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:08,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:08,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:08,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:08,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:08,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:08,728 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-07 01:56:08,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:08,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:08,728 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:08,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:08,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:08,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832628854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832628855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832628856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832628857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832628963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832628963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832628963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:08,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:08,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832628964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,148 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/0c26b85f65384a6db11bec5bf169678b 2023-04-07 01:56:09,167 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/772e48cac7cd4125821f1a7753f7b727 2023-04-07 01:56:09,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832629171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832629171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832629172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832629172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832629482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832629482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832629483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832629484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,583 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f703e089e61e42d2acb048e4b936bace 2023-04-07 01:56:09,587 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/0c26b85f65384a6db11bec5bf169678b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b 2023-04-07 01:56:09,589 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b, entries=150, sequenceid=631, filesize=11.9 K 2023-04-07 01:56:09,590 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/772e48cac7cd4125821f1a7753f7b727 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727 2023-04-07 01:56:09,593 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727, entries=150, sequenceid=631, filesize=11.9 K 2023-04-07 01:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f703e089e61e42d2acb048e4b936bace as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace 2023-04-07 01:56:09,597 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace, entries=150, sequenceid=631, filesize=11.9 K 2023-04-07 01:56:09,597 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 0e0b9000c3ed0a157bd0554455c64d06 in 869ms, sequenceid=631, compaction requested=true 2023-04-07 01:56:09,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:09,597 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:09,599 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:09,599 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f3a1df[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:09,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:09,601 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,603 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:09,603 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,603 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:09,604 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:09,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:09,607 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:09,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:09,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:09,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:09,607 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:09,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:09,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:09,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:09,609 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,615 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:09,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:09,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:09,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:09,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,616 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:09,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:09,616 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:09,716 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:09,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:09,720 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:09,720 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:09,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:09,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:09,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:09,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:09,722 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:09,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:09,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:09,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:09,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:09,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:09,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:09,724 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:09,724 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:09,724 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:09,724 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:09,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:09,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:09,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:09,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:09,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:09,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:09,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:10,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832630019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832570012, max timestamp=1680832570025Num uniques 859; 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832570018, max timestamp=1680832570025Num uniques 859; 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680832570012, max timestamp=1680832570025Num uniques 859; 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:56:10,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 01:56:10,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832630057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832630067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832630068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832630129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,133 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1 2023-04-07 01:56:10,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832630169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832630179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832630179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832630335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832630372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832630386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832630386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37336 deadline: 1680832630430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,436 DEBUG [Thread-3165] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:10,556 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d27067bff7dd431795ac03c8525957db 2023-04-07 01:56:10,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832630641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832630680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832630691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832630693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:10,984 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/eaf95b79906d46cfa294e255b7218130 2023-04-07 01:56:10,987 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1 2023-04-07 01:56:10,990 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1, entries=150, sequenceid=642, filesize=11.9 K 2023-04-07 01:56:10,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/d27067bff7dd431795ac03c8525957db as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db 2023-04-07 01:56:10,994 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db, entries=150, sequenceid=642, filesize=11.9 K 2023-04-07 01:56:10,995 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/eaf95b79906d46cfa294e255b7218130 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130 2023-04-07 01:56:11,000 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130, entries=150, sequenceid=642, filesize=11.9 K 2023-04-07 01:56:11,001 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 0e0b9000c3ed0a157bd0554455c64d06 in 1277ms, sequenceid=642, compaction requested=true 2023-04-07 01:56:11,001 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:11,001 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:11,007 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3992d3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:11,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:11,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,009 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:11,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,010 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:11,011 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:11,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:11,014 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:11,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:11,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:11,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:11,014 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:11,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,015 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,016 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,021 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,021 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:11,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:11,022 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:11,022 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:11,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:11,122 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:11,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:11,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:11,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:11,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:11,127 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:11,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:11,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:11,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,127 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,133 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,133 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,133 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:11,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:11,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:11,136 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=129.71 KB heapSize=365.45 KB 2023-04-07 01:56:11,136 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:11,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:11,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:11,137 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,144 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/95987e0472e4449c951ed731628694d9 2023-04-07 01:56:11,155 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/224ef02bc9d142c68785095a3d07622c 2023-04-07 01:56:11,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:11,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:11,168 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7a6552ebe19144a1aad8d498ed2cce15 2023-04-07 01:56:11,171 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/95987e0472e4449c951ed731628694d9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9 2023-04-07 01:56:11,175 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 01:56:11,175 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/224ef02bc9d142c68785095a3d07622c as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c 2023-04-07 01:56:11,178 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 01:56:11,179 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/7a6552ebe19144a1aad8d498ed2cce15 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15 2023-04-07 01:56:11,183 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 01:56:11,184 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=20.13 KB/20610 for 0e0b9000c3ed0a157bd0554455c64d06 in 48ms, sequenceid=677, compaction requested=true 2023-04-07 01:56:11,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:11,184 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:11,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:11,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:11,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b84e442[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:11,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:11,190 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:11,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:11,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:11,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:11,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:11,204 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:11,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:11,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:11,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:11,204 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:11,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/104fd0aa36fc48e881302147da21a07a 2023-04-07 01:56:11,206 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,208 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,211 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:11,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:11,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,217 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:11,217 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832571232, max timestamp=1680832571247Num uniques 859; 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832571232, max timestamp=1680832571247Num uniques 859; 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680832571232, max timestamp=1680832571247Num uniques 859; 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:11,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 01:56:11,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832631291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832631304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,317 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:11,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:11,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832631307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:11,320 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:11,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:11,320 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:11,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832631307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,323 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:11,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:11,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:11,325 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:11,326 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:11,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@365ac7e6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:11,328 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,330 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:11,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,330 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:11,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:11,333 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:11,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:11,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:11,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:11,333 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:11,334 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,334 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,334 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,336 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,337 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:11,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:11,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,342 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,342 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,342 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832631409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832631409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832631420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832631424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,442 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:11,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:11,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:11,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:11,448 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:11,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:11,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:11,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,449 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,449 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:11,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:11,460 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:11,461 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,461 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:11,461 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:11,461 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,463 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@91875ab[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:11,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:11,466 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,467 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:11,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,468 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:11,472 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:11,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,473 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:11,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:11,475 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:11,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:11,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:11,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:11,475 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:11,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,477 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,478 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,479 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,482 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:11,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,482 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,482 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:11,482 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:11,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,583 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:11,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:11,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:11,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:11,588 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:11,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:11,588 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:11,589 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,589 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,590 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,590 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,591 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:11,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:11,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:11,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:11,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:11,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,592 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:11,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:11,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:11,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,602 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:11,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:11,603 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:11,604 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,605 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:11,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e70d151[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:11,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:11,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:11,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:11,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,610 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:11,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:11,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:11,611 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:11,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:11,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:11,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:11,611 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:11,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,613 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:11,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:11,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,617 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:11,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:11,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:11,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:11,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:11,618 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:11,618 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:11,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/46cf37befbc54738a063d0066ed669f4 2023-04-07 01:56:11,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/77768f985dd44c82ae4f371f883e28ec 2023-04-07 01:56:11,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832631620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832631621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832631630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/104fd0aa36fc48e881302147da21a07a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a 2023-04-07 01:56:11,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a, entries=250, sequenceid=687, filesize=16.7 K 2023-04-07 01:56:11,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/46cf37befbc54738a063d0066ed669f4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4 2023-04-07 01:56:11,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4, entries=150, sequenceid=687, filesize=11.9 K 2023-04-07 01:56:11,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/77768f985dd44c82ae4f371f883e28ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec 2023-04-07 01:56:11,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832631635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec, entries=150, sequenceid=687, filesize=11.9 K 2023-04-07 01:56:11,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 0e0b9000c3ed0a157bd0554455c64d06 in 450ms, sequenceid=687, compaction requested=true 2023-04-07 01:56:11,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:11,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:11,645 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:56:11,645 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:56:11,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:11,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:11,647 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76812 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:56:11,647 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:11,647 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71932 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:56:11,647 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,647 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:11,647 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,647 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/bee7e0bde97d46f0b1c353f63bf905da, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=75.0 K 2023-04-07 01:56:11,647 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/910a7fe7401f46e4a469b7fa6f900373, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=70.2 K 2023-04-07 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting bee7e0bde97d46f0b1c353f63bf905da, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 910a7fe7401f46e4a469b7fa6f900373, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b57ccbe67d654891b5025ab62fcc327c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680832567539 2023-04-07 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d1ed5be6ac154a20bc4871704d94eddf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680832567539 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0c26b85f65384a6db11bec5bf169678b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680832568216 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 772e48cac7cd4125821f1a7753f7b727, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680832568216 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5c9ffa1326cb4cda8ae7ce9dcda64ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680832568853 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d27067bff7dd431795ac03c8525957db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680832568853 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 95987e0472e4449c951ed731628694d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680832570056 2023-04-07 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 224ef02bc9d142c68785095a3d07622c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680832570056 2023-04-07 01:56:11,653 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 104fd0aa36fc48e881302147da21a07a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571177 2023-04-07 01:56:11,653 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 46cf37befbc54738a063d0066ed669f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571183 2023-04-07 01:56:11,662 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#545 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:11,662 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#546 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:11,685 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/d49bbc0131df4b1ea55513fc98f5a812 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d49bbc0131df4b1ea55513fc98f5a812 2023-04-07 01:56:11,690 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into d49bbc0131df4b1ea55513fc98f5a812(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:11,690 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:11,690 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=10, startTime=1680832571645; duration=0sec 2023-04-07 01:56:11,691 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:11,691 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 01:56:11,694 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71932 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 01:56:11,694 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:11,694 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,694 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9ec89dd85c644bbca507e9fc106cea18, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=70.2 K 2023-04-07 01:56:11,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9ec89dd85c644bbca507e9fc106cea18, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680832567203 2023-04-07 01:56:11,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 21364a73822f4600a1d386cc92c1886d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680832567539 2023-04-07 01:56:11,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f703e089e61e42d2acb048e4b936bace, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680832568216 2023-04-07 01:56:11,697 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting eaf95b79906d46cfa294e255b7218130, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680832568853 2023-04-07 01:56:11,697 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7a6552ebe19144a1aad8d498ed2cce15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680832570056 2023-04-07 01:56:11,698 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 77768f985dd44c82ae4f371f883e28ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571183 2023-04-07 01:56:11,709 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#547 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:11,718 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:11,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:11,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:11,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:11,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:11,740 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:11,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:11,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:11,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:11,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:11,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:11,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:11,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:11,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:11,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:11,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:11,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:11,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:11,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:11,755 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:11,755 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. started... 2023-04-07 01:56:11,756 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:11,756 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:11,767 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/21c0a96526974907bf4a5d311ef81e80 2023-04-07 01:56:11,781 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/be09fad194ec4b8b8d0a79b14b4bff0b 2023-04-07 01:56:11,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:11,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. as already flushing 2023-04-07 01:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37302 deadline: 1680832631946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832631947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832631948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:11,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832631954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:12,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37372 deadline: 1680832632055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:12,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37358 deadline: 1680832632059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:12,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37300 deadline: 1680832632060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,077 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/66006093843e4f5aaa1759499b6c00d9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/66006093843e4f5aaa1759499b6c00d9 2023-04-07 01:56:12,081 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into 66006093843e4f5aaa1759499b6c00d9(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:12,081 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:12,081 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=10, startTime=1680832571645; duration=0sec 2023-04-07 01:56:12,081 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:12,128 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/1e33dce933ac46048e9c1e01062d7319 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e33dce933ac46048e9c1e01062d7319 2023-04-07 01:56:12,132 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 1e33dce933ac46048e9c1e01062d7319(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:12,132 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:12,132 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=10, startTime=1680832571645; duration=0sec 2023-04-07 01:56:12,132 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:12,191 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/788b77faa8f847fe8a25f147ed0ad37e 2023-04-07 01:56:12,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/21c0a96526974907bf4a5d311ef81e80 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80 2023-04-07 01:56:12,199 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80, entries=150, sequenceid=724, filesize=11.9 K 2023-04-07 01:56:12,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/be09fad194ec4b8b8d0a79b14b4bff0b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b 2023-04-07 01:56:12,211 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b, entries=150, sequenceid=724, filesize=11.9 K 2023-04-07 01:56:12,212 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/788b77faa8f847fe8a25f147ed0ad37e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e 2023-04-07 01:56:12,216 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e, entries=150, sequenceid=724, filesize=11.9 K 2023-04-07 01:56:12,216 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 0e0b9000c3ed0a157bd0554455c64d06 in 460ms, sequenceid=724, compaction requested=false 2023-04-07 01:56:12,217 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:12,217 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:12,219 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:12,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:12,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32aa9a29[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:12,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:12,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:12,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:12,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:12,225 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:12,226 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:12,231 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:12,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:12,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:12,231 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:12,231 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:12,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,234 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:12,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,234 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:12,235 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:12,235 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:12,238 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:12,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:12,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,239 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:12,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:12,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:12,239 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:12,239 DEBUG [Thread-3173] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:12,239 DEBUG [Thread-3173] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:12,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:12,254 DEBUG [Thread-3182] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:61971 2023-04-07 01:56:12,254 DEBUG [Thread-3176] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:61971 2023-04-07 01:56:12,254 DEBUG [Thread-3182] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,254 DEBUG [Thread-3176] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,254 DEBUG [Thread-3174] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:61971 2023-04-07 01:56:12,254 DEBUG [Thread-3174] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,256 DEBUG [Thread-3178] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:61971 2023-04-07 01:56:12,256 DEBUG [Thread-3178] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,256 DEBUG [Thread-3180] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:61971 2023-04-07 01:56:12,256 DEBUG [Thread-3180] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:12,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:12,262 DEBUG [Thread-3169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:61971 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:12,262 DEBUG [Thread-3169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:12,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:12,265 DEBUG [Thread-3171] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:61971 2023-04-07 01:56:12,265 DEBUG [Thread-3167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:61971 2023-04-07 01:56:12,265 DEBUG [Thread-3171] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,265 DEBUG [Thread-3167] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/67823638a6ab4590bd536bd3d4dea8bf 2023-04-07 01:56:12,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/e47ed0414aad42f1b7a688d73cb24645 2023-04-07 01:56:12,339 DEBUG [Thread-3173] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:12,458 DEBUG [Thread-3163] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:61971 2023-04-07 01:56:12,458 DEBUG [Thread-3163] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:12,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f45fd3acaeb04d12bfee9fcdd8e38131 2023-04-07 01:56:12,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/67823638a6ab4590bd536bd3d4dea8bf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf 2023-04-07 01:56:12,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf, entries=100, sequenceid=736, filesize=9.5 K 2023-04-07 01:56:12,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/e47ed0414aad42f1b7a688d73cb24645 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645 2023-04-07 01:56:12,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645, entries=100, sequenceid=736, filesize=9.5 K 2023-04-07 01:56:12,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f45fd3acaeb04d12bfee9fcdd8e38131 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131 2023-04-07 01:56:12,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131, entries=100, sequenceid=736, filesize=9.5 K 2023-04-07 01:56:12,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for 0e0b9000c3ed0a157bd0554455c64d06 in 432ms, sequenceid=736, compaction requested=true 2023-04-07 01:56:12,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:12,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:12,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:12,694 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:12,694 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:12,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:12,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:12,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/A is initiating minor compaction (all files) 2023-04-07 01:56:12,695 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:12,695 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/A in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:12,695 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/B is initiating minor compaction (all files) 2023-04-07 01:56:12,695 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d49bbc0131df4b1ea55513fc98f5a812, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.7 K 2023-04-07 01:56:12,695 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/B in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:12,695 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/66006093843e4f5aaa1759499b6c00d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.7 K 2023-04-07 01:56:12,695 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d49bbc0131df4b1ea55513fc98f5a812, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571183 2023-04-07 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 66006093843e4f5aaa1759499b6c00d9, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571183 2023-04-07 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 21c0a96526974907bf4a5d311ef81e80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1680832571289 2023-04-07 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting be09fad194ec4b8b8d0a79b14b4bff0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1680832571289 2023-04-07 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e47ed0414aad42f1b7a688d73cb24645, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1680832571953 2023-04-07 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 67823638a6ab4590bd536bd3d4dea8bf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1680832571953 2023-04-07 01:56:12,701 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#B#compaction#554 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:12,701 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#A#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:13,108 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/4318a539daed4caabed247c3f412d9bd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4318a539daed4caabed247c3f412d9bd 2023-04-07 01:56:13,109 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/a125d65622ed4d33abd173c9ceaab363 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a125d65622ed4d33abd173c9ceaab363 2023-04-07 01:56:13,113 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/A of 0e0b9000c3ed0a157bd0554455c64d06 into 4318a539daed4caabed247c3f412d9bd(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:13,113 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/B of 0e0b9000c3ed0a157bd0554455c64d06 into a125d65622ed4d33abd173c9ceaab363(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:13,113 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:13,113 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/A, priority=13, startTime=1680832572694; duration=0sec 2023-04-07 01:56:13,113 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:13,113 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/B, priority=13, startTime=1680832572694; duration=0sec 2023-04-07 01:56:13,113 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:13,113 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:13,113 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:13,115 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:13,115 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 0e0b9000c3ed0a157bd0554455c64d06/C is initiating minor compaction (all files) 2023-04-07 01:56:13,115 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0e0b9000c3ed0a157bd0554455c64d06/C in TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:13,116 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e33dce933ac46048e9c1e01062d7319, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp, totalSize=34.7 K 2023-04-07 01:56:13,116 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1e33dce933ac46048e9c1e01062d7319, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680832571183 2023-04-07 01:56:13,116 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 788b77faa8f847fe8a25f147ed0ad37e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1680832571289 2023-04-07 01:56:13,117 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f45fd3acaeb04d12bfee9fcdd8e38131, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1680832571953 2023-04-07 01:56:13,124 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0e0b9000c3ed0a157bd0554455c64d06#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:13,532 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/0d329403d2ce42e491e42b5219d26eef as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/0d329403d2ce42e491e42b5219d26eef 2023-04-07 01:56:13,535 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0e0b9000c3ed0a157bd0554455c64d06/C of 0e0b9000c3ed0a157bd0554455c64d06 into 0d329403d2ce42e491e42b5219d26eef(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:13,535 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:13,535 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06., storeName=0e0b9000c3ed0a157bd0554455c64d06/C, priority=13, startTime=1680832572694; duration=0sec 2023-04-07 01:56:13,535 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:20,464 DEBUG [Thread-3165] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:61971 2023-04-07 01:56:20,464 DEBUG [Thread-3165] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:20,464 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:56:20,464 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-07 01:56:20,464 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2108 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6321 rows 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2102 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6306 rows 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2109 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6323 rows 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2113 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6339 rows 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(402): scanned 2090 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(403): verified 6270 rows 2023-04-07 01:56:20,465 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:56:20,465 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a987907 to 127.0.0.1:61971 2023-04-07 01:56:20,465 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:20,469 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:56:20,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:56:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:20,473 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832580473"}]},"ts":"1680832580473"} 2023-04-07 01:56:20,474 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:56:20,476 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:56:20,477 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, UNASSIGN}] 2023-04-07 01:56:20,478 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, UNASSIGN 2023-04-07 01:56:20,478 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0e0b9000c3ed0a157bd0554455c64d06, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:20,478 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832580478"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832580478"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832580478"}]},"ts":"1680832580478"} 2023-04-07 01:56:20,479 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:56:20,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:20,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0e0b9000c3ed0a157bd0554455c64d06, disabling compactions & flushes 2023-04-07 01:56:20,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. after waiting 0 ms 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:20,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0e0b9000c3ed0a157bd0554455c64d06 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=A 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=B 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0e0b9000c3ed0a157bd0554455c64d06, store=C 2023-04-07 01:56:20,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:21,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c73fa50b79f44842aa806337037f1efb 2023-04-07 01:56:21,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/5edad869337c40f29ad83f2fa402284a 2023-04-07 01:56:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:21,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f75146e7c8ee43f5b66d355b992ca5dd 2023-04-07 01:56:21,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/A/c73fa50b79f44842aa806337037f1efb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c73fa50b79f44842aa806337037f1efb 2023-04-07 01:56:21,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c73fa50b79f44842aa806337037f1efb, entries=100, sequenceid=746, filesize=9.5 K 2023-04-07 01:56:21,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/B/5edad869337c40f29ad83f2fa402284a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5edad869337c40f29ad83f2fa402284a 2023-04-07 01:56:21,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5edad869337c40f29ad83f2fa402284a, entries=100, sequenceid=746, filesize=9.5 K 2023-04-07 01:56:21,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/.tmp/C/f75146e7c8ee43f5b66d355b992ca5dd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f75146e7c8ee43f5b66d355b992ca5dd 2023-04-07 01:56:21,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f75146e7c8ee43f5b66d355b992ca5dd, entries=100, sequenceid=746, filesize=9.5 K 2023-04-07 01:56:21,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0e0b9000c3ed0a157bd0554455c64d06 in 853ms, sequenceid=746, compaction requested=false 2023-04-07 01:56:21,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b3e860719d144f0d8f28e8f7b270fd82, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8405f54396654abcb2d837405acffc63, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/80ad28af6a5446259f8c8e5a077a1e27, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/a53b248b059c455ba4db131c95e33d54, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/60b35d9f1dfb4f369192e2a3119781d2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ccd842f7976459abfe37d5d68fd5b7a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b0ab678475cf4c5c99f3323f23462ae5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9a4b046f2ea9478e848783616192de94, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b96c4bf5d2f24ff9955a8850693899a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d9275ec5214f40e4a4d29e3f9b0d4d48, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/bee7e0bde97d46f0b1c353f63bf905da, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d49bbc0131df4b1ea55513fc98f5a812, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf] to archive 2023-04-07 01:56:21,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:21,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/23d82d8fb1e44a62816d184453d887f5 2023-04-07 01:56:21,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9286ffad180e40c49d13a80077b4cad4 2023-04-07 01:56:21,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c1a3d0895d85406c9bc2659f3b0aa192 2023-04-07 01:56:21,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b3e860719d144f0d8f28e8f7b270fd82 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b3e860719d144f0d8f28e8f7b270fd82 2023-04-07 01:56:21,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d86511f998c247af8f68754e7d181069 2023-04-07 01:56:21,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/07386df652284fe7ac3d432745595a01 2023-04-07 01:56:21,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8405f54396654abcb2d837405acffc63 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8405f54396654abcb2d837405acffc63 2023-04-07 01:56:21,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ac7e33cf606450f9733e73f18c1750f 2023-04-07 01:56:21,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/6a0d35876dc242f08a425cdabc45068c 2023-04-07 01:56:21,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/80ad28af6a5446259f8c8e5a077a1e27 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/80ad28af6a5446259f8c8e5a077a1e27 2023-04-07 01:56:21,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/ddb78f730fb04e26be8bc4efe8328348 2023-04-07 01:56:21,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b2fb33affac94116af122fd23533ea4a 2023-04-07 01:56:21,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/a53b248b059c455ba4db131c95e33d54 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/a53b248b059c455ba4db131c95e33d54 2023-04-07 01:56:21,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4a25070e21af427a8275a0d10cacad25 2023-04-07 01:56:21,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/60b35d9f1dfb4f369192e2a3119781d2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/60b35d9f1dfb4f369192e2a3119781d2 2023-04-07 01:56:21,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/959de7cf822a40acb764e093437e67eb 2023-04-07 01:56:21,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/f00903e1f3e14044afc366f7c20e7d01 2023-04-07 01:56:21,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/8a9d5b5f0b0e4be2912374e0b09a0252 2023-04-07 01:56:21,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/49bb811442924b7e9376c5ab71a88ac6 2023-04-07 01:56:21,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ccd842f7976459abfe37d5d68fd5b7a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/2ccd842f7976459abfe37d5d68fd5b7a 2023-04-07 01:56:21,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/7ebf2b7dc18e467281b09907e818b3e3 2023-04-07 01:56:21,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/eb064f1e1d6d4ab4bd0c248c0177b895 2023-04-07 01:56:21,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b0ab678475cf4c5c99f3323f23462ae5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b0ab678475cf4c5c99f3323f23462ae5 2023-04-07 01:56:21,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9c008554c0214fd6af08db1476ec3293 2023-04-07 01:56:21,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0b8f2ccdff744ac8b1e7d036ff2aefb8 2023-04-07 01:56:21,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/06eba8ca9190466aac3bbdf0036e8c78 2023-04-07 01:56:21,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9a4b046f2ea9478e848783616192de94 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9a4b046f2ea9478e848783616192de94 2023-04-07 01:56:21,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/814558d09c4f471f86324cc73ad3ddec 2023-04-07 01:56:21,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9af191be1f5c40838538d4c895925811 2023-04-07 01:56:21,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b96c4bf5d2f24ff9955a8850693899a3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b96c4bf5d2f24ff9955a8850693899a3 2023-04-07 01:56:21,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c0f004a916814fc39525473b08903fe8 2023-04-07 01:56:21,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/9db45593624c449fbc276fe3be8ce586 2023-04-07 01:56:21,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b9fbd69894c84aba852dd19d2906b4a3 2023-04-07 01:56:21,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d9275ec5214f40e4a4d29e3f9b0d4d48 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d9275ec5214f40e4a4d29e3f9b0d4d48 2023-04-07 01:56:21,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/dc8ea3404c3b4ceaa1a47da1245b2f03 2023-04-07 01:56:21,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b37220456711490bac2fc46d90b5fb28 2023-04-07 01:56:21,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/e5a935e713dd4b75a2d693f332017ae5 2023-04-07 01:56:21,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/bee7e0bde97d46f0b1c353f63bf905da to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/bee7e0bde97d46f0b1c353f63bf905da 2023-04-07 01:56:21,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/b57ccbe67d654891b5025ab62fcc327c 2023-04-07 01:56:21,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/0c26b85f65384a6db11bec5bf169678b 2023-04-07 01:56:21,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/5c9ffa1326cb4cda8ae7ce9dcda64ce1 2023-04-07 01:56:21,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/95987e0472e4449c951ed731628694d9 2023-04-07 01:56:21,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/104fd0aa36fc48e881302147da21a07a 2023-04-07 01:56:21,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d49bbc0131df4b1ea55513fc98f5a812 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/d49bbc0131df4b1ea55513fc98f5a812 2023-04-07 01:56:21,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/21c0a96526974907bf4a5d311ef81e80 2023-04-07 01:56:21,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/67823638a6ab4590bd536bd3d4dea8bf 2023-04-07 01:56:21,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0f1e5b66a25148e195d5fc74ac3790f4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/54182927106d407ab25b19b973b9814a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/fb748c2af82344b19d75059db122bc96, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/f3e165fed8804dcab1ce74a36f52096e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/16f16f8060494a67b469d145e175ad87, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/9f57caf3b77e4d29aa086c7df646b79d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d7575f9555fa4dc2927da8cc6aa73e7c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5497a4eaf6a44542942e4b0cd67dac16, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e33fcecd4e6c428f9558a659d75a12a2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a6aad2871caa4df389bb2528effe7cd3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/910a7fe7401f46e4a469b7fa6f900373, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/66006093843e4f5aaa1759499b6c00d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645] to archive 2023-04-07 01:56:21,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:21,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d995079b56f04b4fa741acada230381c 2023-04-07 01:56:21,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0da6d0a7868940d69d5ac4b461f79265 2023-04-07 01:56:21,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0f1e5b66a25148e195d5fc74ac3790f4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0f1e5b66a25148e195d5fc74ac3790f4 2023-04-07 01:56:21,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/caa3974421a94421a7634444f738f7c0 2023-04-07 01:56:21,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/989ab59ff40a4888990526e968ebb5a6 2023-04-07 01:56:21,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/54182927106d407ab25b19b973b9814a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/54182927106d407ab25b19b973b9814a 2023-04-07 01:56:21,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/acf21b587764411e8dcae548e890a738 2023-04-07 01:56:21,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ec6cb31d8ff946d28b24947825c410c0 2023-04-07 01:56:21,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/fb748c2af82344b19d75059db122bc96 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/fb748c2af82344b19d75059db122bc96 2023-04-07 01:56:21,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/319b1879bb51494cbabf7cb16dbc51be 2023-04-07 01:56:21,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/40cb428a8ed148c7a8a41008fcf35613 2023-04-07 01:56:21,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/f3e165fed8804dcab1ce74a36f52096e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/f3e165fed8804dcab1ce74a36f52096e 2023-04-07 01:56:21,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0a1257ecb73f4b4e8d67f387857c6622 2023-04-07 01:56:21,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/c80afae854c6444ab959bf5716487ab3 2023-04-07 01:56:21,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/16f16f8060494a67b469d145e175ad87 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/16f16f8060494a67b469d145e175ad87 2023-04-07 01:56:21,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7485ac6afa3d4c6eba98b54b1b2ee40c 2023-04-07 01:56:21,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/2c727ac98e3a4b2ea3872e958d582a63 2023-04-07 01:56:21,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/35ab8a06eccc4b5e89f1053ea3e40bc8 2023-04-07 01:56:21,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/9f57caf3b77e4d29aa086c7df646b79d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/9f57caf3b77e4d29aa086c7df646b79d 2023-04-07 01:56:21,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/78d156cfbb9b4e6fab20b5279b976d87 2023-04-07 01:56:21,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7d3cf132ac3d41c18f5c8bbd7006904c 2023-04-07 01:56:21,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d7575f9555fa4dc2927da8cc6aa73e7c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d7575f9555fa4dc2927da8cc6aa73e7c 2023-04-07 01:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:21,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/aa4dd55268ff45ca8243a0261f22638f 2023-04-07 01:56:21,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/1b340ea7202548599905d1f4c17d7898 2023-04-07 01:56:21,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/0b86e70274e9466ebb27b1c7fb8da0b0 2023-04-07 01:56:21,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5497a4eaf6a44542942e4b0cd67dac16 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5497a4eaf6a44542942e4b0cd67dac16 2023-04-07 01:56:21,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/befb389f756b438095f6d22cb946a105 2023-04-07 01:56:21,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/3c0efc82d4b440d392fdad0cc8532a33 2023-04-07 01:56:21,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/65f5f73a62e6435abd1b50092e74eeda 2023-04-07 01:56:21,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e33fcecd4e6c428f9558a659d75a12a2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e33fcecd4e6c428f9558a659d75a12a2 2023-04-07 01:56:21,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/7a9df67f281f474ea716f1a7409d3c96 2023-04-07 01:56:21,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d6efe7e6793e499b88617b6fe30fe927 2023-04-07 01:56:21,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a6aad2871caa4df389bb2528effe7cd3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a6aad2871caa4df389bb2528effe7cd3 2023-04-07 01:56:21,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/ae2aa76924e84653abe3d8a6795baef6 2023-04-07 01:56:21,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/00d0ca18cac24fd8a7b52a7809c57194 2023-04-07 01:56:21,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/815a1df574ea4f0697be157bb4234ef6 2023-04-07 01:56:21,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/910a7fe7401f46e4a469b7fa6f900373 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/910a7fe7401f46e4a469b7fa6f900373 2023-04-07 01:56:21,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/04df03f02df7417e92a8610b50ecb5d9 2023-04-07 01:56:21,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d1ed5be6ac154a20bc4871704d94eddf 2023-04-07 01:56:21,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/772e48cac7cd4125821f1a7753f7b727 2023-04-07 01:56:21,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/d27067bff7dd431795ac03c8525957db 2023-04-07 01:56:21,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/224ef02bc9d142c68785095a3d07622c 2023-04-07 01:56:21,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/66006093843e4f5aaa1759499b6c00d9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/66006093843e4f5aaa1759499b6c00d9 2023-04-07 01:56:21,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/46cf37befbc54738a063d0066ed669f4 2023-04-07 01:56:21,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/be09fad194ec4b8b8d0a79b14b4bff0b 2023-04-07 01:56:21,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/e47ed0414aad42f1b7a688d73cb24645 2023-04-07 01:56:21,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/272289ccc6ba4f8cbb89923ca0066ebf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2c1d301a742f472fa68e8cc73c573d64, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ecbefac8940d432c8314dda7c34e70e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7bfa18f83cc44bd38c47a53568ee6a3c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/526833cf71c141eaa83b713c8df6d8ee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e077ac84afb46a8b2673fcc49f33970, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/94a1c555c3364e94bddd35ee8f905889, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a70db1fc9e44df2b833b43cbe89686d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/c0b28de473e34fda9df700de4bd9c07a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9ec89dd85c644bbca507e9fc106cea18, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e33dce933ac46048e9c1e01062d7319, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131] to archive 2023-04-07 01:56:21,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:21,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f18e1bacf90a470faa7e79575aea410a 2023-04-07 01:56:21,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3805ff39524b4da584a1eada245530e4 2023-04-07 01:56:21,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/272289ccc6ba4f8cbb89923ca0066ebf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/272289ccc6ba4f8cbb89923ca0066ebf 2023-04-07 01:56:21,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a38fcac604404274972f728933d9444d 2023-04-07 01:56:21,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d6bec5b80dec4859ab92223128813137 2023-04-07 01:56:21,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2c1d301a742f472fa68e8cc73c573d64 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2c1d301a742f472fa68e8cc73c573d64 2023-04-07 01:56:21,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/33036ab51de443b288e5edbeb4cf46a0 2023-04-07 01:56:21,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f530c3ecae174e40ae30645bdc4c0500 2023-04-07 01:56:21,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ecbefac8940d432c8314dda7c34e70e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ecbefac8940d432c8314dda7c34e70e5 2023-04-07 01:56:21,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7508e4fe6e224d80937f0399de6640fd 2023-04-07 01:56:21,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3e6e171b289d4431a92ba169e187ce13 2023-04-07 01:56:21,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7f58ec8af8a0489b9513e179ad41124b 2023-04-07 01:56:21,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7bfa18f83cc44bd38c47a53568ee6a3c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7bfa18f83cc44bd38c47a53568ee6a3c 2023-04-07 01:56:21,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/25f5c8ab5800451f90fc235d3300a3f2 2023-04-07 01:56:21,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/11738473a0464acc8ec4e0cf47676c15 2023-04-07 01:56:21,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/526833cf71c141eaa83b713c8df6d8ee to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/526833cf71c141eaa83b713c8df6d8ee 2023-04-07 01:56:21,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/4419abf9612942a18b4b03e59684a007 2023-04-07 01:56:21,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/53d94e4232544593abf9db246925b29c 2023-04-07 01:56:21,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/ba6fe777d0ba421d9ad7a4d0d1cab4b7 2023-04-07 01:56:21,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/31d45437babf4720a47171ec289ed069 2023-04-07 01:56:21,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3d59fdcb07aa43f0824f79fe2144f7a3 2023-04-07 01:56:21,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e077ac84afb46a8b2673fcc49f33970 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e077ac84afb46a8b2673fcc49f33970 2023-04-07 01:56:21,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2b5f733b237a4f32aa9c35a378ddb59a 2023-04-07 01:56:21,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/d41029154e4b45fc80afe2b63ca960fc 2023-04-07 01:56:21,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/5d4b37254dfa4c9da3b3df221183986d 2023-04-07 01:56:21,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/94a1c555c3364e94bddd35ee8f905889 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/94a1c555c3364e94bddd35ee8f905889 2023-04-07 01:56:21,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/3330c376867247328d06c044c50fd1ac 2023-04-07 01:56:21,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/6e929d3dba8f44ada1b2e0d6572f676a 2023-04-07 01:56:21,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a70db1fc9e44df2b833b43cbe89686d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a70db1fc9e44df2b833b43cbe89686d 2023-04-07 01:56:21,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/667cd7d13add47499910db2d606ee216 2023-04-07 01:56:21,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/28f7f485b374463b92373008640b39b1 2023-04-07 01:56:21,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/a7d23a64e619417bbea9bf0136c9fbee 2023-04-07 01:56:21,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/c0b28de473e34fda9df700de4bd9c07a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/c0b28de473e34fda9df700de4bd9c07a 2023-04-07 01:56:21,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9425e7f05bfa4a7db4d749b19358ef7b 2023-04-07 01:56:21,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/83e5f2bd42a44c3399425c8d04cf10e5 2023-04-07 01:56:21,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f51133eb3196455f847a9cc48328a5d9 2023-04-07 01:56:21,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9ec89dd85c644bbca507e9fc106cea18 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/9ec89dd85c644bbca507e9fc106cea18 2023-04-07 01:56:21,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/2ec64f8389e34059be7a530c3a2ff416 2023-04-07 01:56:21,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/21364a73822f4600a1d386cc92c1886d 2023-04-07 01:56:21,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f703e089e61e42d2acb048e4b936bace 2023-04-07 01:56:21,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/eaf95b79906d46cfa294e255b7218130 2023-04-07 01:56:21,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/7a6552ebe19144a1aad8d498ed2cce15 2023-04-07 01:56:21,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e33dce933ac46048e9c1e01062d7319 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/1e33dce933ac46048e9c1e01062d7319 2023-04-07 01:56:21,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/77768f985dd44c82ae4f371f883e28ec 2023-04-07 01:56:21,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/788b77faa8f847fe8a25f147ed0ad37e 2023-04-07 01:56:21,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f45fd3acaeb04d12bfee9fcdd8e38131 2023-04-07 01:56:21,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/recovered.edits/749.seqid, newMaxSeqId=749, maxSeqId=1 2023-04-07 01:56:21,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06. 2023-04-07 01:56:21,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0e0b9000c3ed0a157bd0554455c64d06: 2023-04-07 01:56:21,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:21,641 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0e0b9000c3ed0a157bd0554455c64d06, regionState=CLOSED 2023-04-07 01:56:21,641 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832581641"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832581641"}]},"ts":"1680832581641"} 2023-04-07 01:56:21,643 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-07 01:56:21,643 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 0e0b9000c3ed0a157bd0554455c64d06, server=jenkins-hbase4.apache.org,44961,1680832421898 in 1.1630 sec 2023-04-07 01:56:21,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-07 01:56:21,645 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0e0b9000c3ed0a157bd0554455c64d06, UNASSIGN in 1.1660 sec 2023-04-07 01:56:21,645 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832581645"}]},"ts":"1680832581645"} 2023-04-07 01:56:21,646 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:56:21,647 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:56:21,649 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1780 sec 2023-04-07 01:56:22,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 01:56:22,577 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-07 01:56:22,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:56:22,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,579 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,580 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-07 01:56:22,586 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:22,590 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/recovered.edits] 2023-04-07 01:56:22,594 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4318a539daed4caabed247c3f412d9bd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/4318a539daed4caabed247c3f412d9bd 2023-04-07 01:56:22,596 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c73fa50b79f44842aa806337037f1efb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/A/c73fa50b79f44842aa806337037f1efb 2023-04-07 01:56:22,598 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5edad869337c40f29ad83f2fa402284a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/5edad869337c40f29ad83f2fa402284a 2023-04-07 01:56:22,600 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a125d65622ed4d33abd173c9ceaab363 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/B/a125d65622ed4d33abd173c9ceaab363 2023-04-07 01:56:22,602 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/0d329403d2ce42e491e42b5219d26eef to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/0d329403d2ce42e491e42b5219d26eef 2023-04-07 01:56:22,603 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f75146e7c8ee43f5b66d355b992ca5dd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/C/f75146e7c8ee43f5b66d355b992ca5dd 2023-04-07 01:56:22,606 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/recovered.edits/749.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06/recovered.edits/749.seqid 2023-04-07 01:56:22,607 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/0e0b9000c3ed0a157bd0554455c64d06 2023-04-07 01:56:22,607 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:56:22,609 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,610 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:56:22,611 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:56:22,612 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,612 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:56:22,612 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832582612"}]},"ts":"9223372036854775807"} 2023-04-07 01:56:22,614 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:56:22,614 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0e0b9000c3ed0a157bd0554455c64d06, NAME => 'TestAcidGuarantees,,1680832551084.0e0b9000c3ed0a157bd0554455c64d06.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:56:22,614 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:56:22,614 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832582614"}]},"ts":"9223372036854775807"} 2023-04-07 01:56:22,615 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:56:22,617 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,618 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-07 01:56:22,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-07 01:56:22,683 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-07 01:56:22,695 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=549 (was 547) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=359 (was 300) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2671 (was 2509) - AvailableMemoryMB LEAK? - 2023-04-07 01:56:22,707 INFO [Listener at localhost/45521] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=549, MaxFileDescriptor=60000, SystemLoadAverage=359, ProcessCount=169, AvailableMemoryMB=2670 2023-04-07 01:56:22,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:56:22,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:56:22,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:22,712 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 01:56:22,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-07 01:56:22,712 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 01:56:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:22,713 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:22,714 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 empty. 2023-04-07 01:56:22,714 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:22,714 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:56:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:23,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:23,120 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:56:23,122 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2c0ce65cbec62d633a29a3f2a06a7a49, NAME => 'TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp 2023-04-07 01:56:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2c0ce65cbec62d633a29a3f2a06a7a49, disabling compactions & flushes 2023-04-07 01:56:23,527 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. after waiting 0 ms 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,527 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,527 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:23,529 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 01:56:23,530 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832583529"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832583529"}]},"ts":"1680832583529"} 2023-04-07 01:56:23,530 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 01:56:23,531 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 01:56:23,531 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832583531"}]},"ts":"1680832583531"} 2023-04-07 01:56:23,532 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 01:56:23,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, ASSIGN}] 2023-04-07 01:56:23,536 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, ASSIGN 2023-04-07 01:56:23,536 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=false 2023-04-07 01:56:23,688 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:23,688 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832583688"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832583688"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832583688"}]},"ts":"1680832583688"} 2023-04-07 01:56:23,689 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:56:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:23,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2c0ce65cbec62d633a29a3f2a06a7a49, NAME => 'TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:56:23,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:56:23,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,845 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,846 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A 2023-04-07 01:56:23,846 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A 2023-04-07 01:56:23,846 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:23,847 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName A 2023-04-07 01:56:23,847 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:23,847 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,848 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B 2023-04-07 01:56:23,848 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B 2023-04-07 01:56:23,848 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:23,849 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName B 2023-04-07 01:56:23,849 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:23,849 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,850 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C 2023-04-07 01:56:23,850 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C 2023-04-07 01:56:23,850 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:23,850 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName C 2023-04-07 01:56:23,851 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:23,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:56:23,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:23,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 01:56:23,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2c0ce65cbec62d633a29a3f2a06a7a49; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62465705, jitterRate=-0.06918846070766449}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:56:23,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:23,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., pid=56, masterSystemTime=1680832583840 2023-04-07 01:56:23,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:23,857 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:23,857 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832583857"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832583857"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832583857"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832583857"}]},"ts":"1680832583857"} 2023-04-07 01:56:23,859 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-07 01:56:23,859 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 in 169 msec 2023-04-07 01:56:23,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-07 01:56:23,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, ASSIGN in 325 msec 2023-04-07 01:56:23,860 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 01:56:23,861 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832583860"}]},"ts":"1680832583860"} 2023-04-07 01:56:23,861 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 01:56:23,863 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 01:56:23,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-04-07 01:56:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 01:56:24,817 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-07 01:56:24,817 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x50529bf3 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:24,822 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bafb41f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:24,823 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:24,824 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:24,825 DEBUG [Listener at localhost/45521] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 01:56:24,827 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 01:56:24,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 01:56:24,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 01:56:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:25,237 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:56:25,238 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 01:56:25,238 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 01:56:25,240 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 01:56:25,242 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, REOPEN/MOVE}] 2023-04-07 01:56:25,242 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, REOPEN/MOVE 2023-04-07 01:56:25,243 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,243 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832585243"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832585243"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832585243"}]},"ts":"1680832585243"} 2023-04-07 01:56:25,244 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:56:25,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2c0ce65cbec62d633a29a3f2a06a7a49, disabling compactions & flushes 2023-04-07 01:56:25,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. after waiting 0 ms 2023-04-07 01:56:25,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 01:56:25,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:25,400 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2c0ce65cbec62d633a29a3f2a06a7a49 to self. 2023-04-07 01:56:25,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,401 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=CLOSED 2023-04-07 01:56:25,401 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832585401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832585401"}]},"ts":"1680832585401"} 2023-04-07 01:56:25,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-07 01:56:25,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 in 158 msec 2023-04-07 01:56:25,404 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44961,1680832421898; forceNewPlan=false, retain=true 2023-04-07 01:56:25,554 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,554 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832585554"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832585554"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832585554"}]},"ts":"1680832585554"} 2023-04-07 01:56:25,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:56:25,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2c0ce65cbec62d633a29a3f2a06a7a49, NAME => 'TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.', STARTKEY => '', ENDKEY => ''} 2023-04-07 01:56:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 01:56:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,713 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,714 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A 2023-04-07 01:56:25,715 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A 2023-04-07 01:56:25,715 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:25,715 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName A 2023-04-07 01:56:25,716 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:25,717 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,719 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B 2023-04-07 01:56:25,719 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B 2023-04-07 01:56:25,719 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:25,719 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName B 2023-04-07 01:56:25,720 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:25,720 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,721 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C 2023-04-07 01:56:25,721 DEBUG [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C 2023-04-07 01:56:25,721 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 01:56:25,721 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c0ce65cbec62d633a29a3f2a06a7a49 columnFamilyName C 2023-04-07 01:56:25,722 INFO [StoreOpener-2c0ce65cbec62d633a29a3f2a06a7a49-1] regionserver.HStore(310): Store=2c0ce65cbec62d633a29a3f2a06a7a49/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 01:56:25,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 01:56:25,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2c0ce65cbec62d633a29a3f2a06a7a49; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74622608, jitterRate=0.11196351051330566}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 01:56:25,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:25,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., pid=61, masterSystemTime=1680832585707 2023-04-07 01:56:25,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,747 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,747 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832585747"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680832585747"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680832585747"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680832585747"}]},"ts":"1680832585747"} 2023-04-07 01:56:25,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-07 01:56:25,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 in 193 msec 2023-04-07 01:56:25,750 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-07 01:56:25,750 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, REOPEN/MOVE in 507 msec 2023-04-07 01:56:25,751 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-07 01:56:25,751 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 510 msec 2023-04-07 01:56:25,752 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 924 msec 2023-04-07 01:56:25,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-07 01:56:25,754 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,759 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,760 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,769 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,769 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,779 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,780 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,795 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,796 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,807 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,808 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,816 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,816 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,824 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,824 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,835 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,835 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,843 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,843 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:61971 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 01:56:25,856 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 01:56:25,874 DEBUG [hconnection-0xf96eeda-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,879 DEBUG [hconnection-0x39cbd810-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,879 DEBUG [hconnection-0x466459d7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,879 DEBUG [hconnection-0x7f3f3874-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,880 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,884 DEBUG [hconnection-0x79f540d5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,884 DEBUG [hconnection-0x3c81b7eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,885 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,897 DEBUG [hconnection-0x5e593958-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,897 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,897 DEBUG [hconnection-0x2024450-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,897 DEBUG [hconnection-0x300ced91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,897 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,898 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:25,906 DEBUG [hconnection-0x56194d37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 01:56:25,907 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,907 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,907 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,907 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 01:56:25,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:25,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:25,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:25,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:25,922 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:25,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:25,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:25,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:25,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,922 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:25,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,923 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:25,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:25,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,924 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:25,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:25,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:25,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:25,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:25,925 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,925 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:25,925 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:25,925 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:25,925 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:25,927 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:25,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,927 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:25,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af1e6d[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:25,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:25,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:25,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:25,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:25,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:25,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:25,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:25,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:25,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:25,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:25,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:25,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832645935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832645935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832645937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832645937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:25,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:25,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832645938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:25,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,942 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407399ef0b8d3784c63a0a793639231c038 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407399ef0b8d3784c63a0a793639231c038 2023-04-07 01:56:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:25,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:25,946 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:25,946 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:25,946 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:25,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:25,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:25,946 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:25,946 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:25,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:25,948 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:25,974 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/f56f0a93b92c42de8ab7cc7f59e115ec 2023-04-07 01:56:26,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/606615652abd41f4868e6229f4b3d2bf 2023-04-07 01:56:26,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832646039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832646040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832646040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832646040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832646041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,046 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:26,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:26,049 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:26,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:26,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:26,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,052 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:26,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:26,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:26,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:26,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:26,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:26,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:26,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,056 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d09cbe[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:26,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:26,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,059 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:26,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,059 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:26,061 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:26,063 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:26,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:26,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:26,063 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:26,063 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:26,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,074 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,074 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:26,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:26,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:26,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,074 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:26,074 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:26,174 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:26,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:26,177 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:26,177 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:26,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:26,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,179 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,179 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:26,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:26,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:26,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:26,189 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:26,189 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:26,190 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:26,190 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:26,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50623cac[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:26,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,196 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:26,197 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:26,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:26,200 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:26,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:26,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:26,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:26,200 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:26,202 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,202 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,202 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,203 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,207 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,207 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,207 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:26,208 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:26,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832646241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832646243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832646244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832646244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832646244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,308 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:26,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:26,311 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:26,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:26,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:26,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,312 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,314 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,314 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:26,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:26,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ff9f5b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:26,319 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:26,319 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:26,321 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:26,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:26,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,322 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:26,322 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,322 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:26,324 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:26,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:26,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:26,324 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:26,324 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:26,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,327 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,327 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,329 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:26,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:26,331 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:26,332 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,332 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,332 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,332 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:26,332 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:26,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/a1c5eec9e52442fab626a196a525523e 2023-04-07 01:56:26,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/f56f0a93b92c42de8ab7cc7f59e115ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec 2023-04-07 01:56:26,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec, entries=150, sequenceid=17, filesize=25.1 K 2023-04-07 01:56:26,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/606615652abd41f4868e6229f4b3d2bf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf 2023-04-07 01:56:26,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf, entries=150, sequenceid=17, filesize=11.6 K 2023-04-07 01:56:26,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/a1c5eec9e52442fab626a196a525523e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e 2023-04-07 01:56:26,432 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:26,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e, entries=150, sequenceid=17, filesize=11.6 K 2023-04-07 01:56:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:26,433 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 512ms, sequenceid=17, compaction requested=false 2023-04-07 01:56:26,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:26,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:26,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:26,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:26,435 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:26,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:26,435 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:26,436 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,436 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,437 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:26,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:26,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:26,439 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:26,439 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:26,452 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304076d732cd619ea4513836360b2c639e65b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076d732cd619ea4513836360b2c639e65b 2023-04-07 01:56:26,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:26,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:26,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832646551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832646553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832646554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832646554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832646555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832646655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832646656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832646674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832646675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832646675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,857 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/adfe3af564324f0184f749376411e1e5 2023-04-07 01:56:26,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832646860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832646860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,866 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/0bb79ea0e34f47ba9cfc137ba4bff7c1 2023-04-07 01:56:26,875 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/cb9c76d99b0d4361ad4199aa0b95c1ed 2023-04-07 01:56:26,878 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/adfe3af564324f0184f749376411e1e5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5 2023-04-07 01:56:26,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832646877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832646877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832646878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,882 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5, entries=150, sequenceid=36, filesize=25.1 K 2023-04-07 01:56:26,883 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/0bb79ea0e34f47ba9cfc137ba4bff7c1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1 2023-04-07 01:56:26,885 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1, entries=150, sequenceid=36, filesize=11.6 K 2023-04-07 01:56:26,886 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/cb9c76d99b0d4361ad4199aa0b95c1ed as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed 2023-04-07 01:56:26,889 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed, entries=150, sequenceid=36, filesize=11.6 K 2023-04-07 01:56:26,889 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 450ms, sequenceid=36, compaction requested=false 2023-04-07 01:56:26,890 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:26,890 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,891 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c09a39f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:26,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:26,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,895 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:26,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,897 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:26,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:26,899 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:26,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:26,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:26,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:26,899 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:26,901 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,901 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:26,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,901 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:26,901 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:26,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:26,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:26,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:26,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:26,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:26,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:26,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:26,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:26,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:26,905 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:26,905 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:26,905 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,005 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:27,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:27,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:27,008 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:27,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:27,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:27,008 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:27,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,009 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,012 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:27,012 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:27,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:27,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:27,014 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:27,014 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:27,014 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:56:27,014 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:27,015 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:27,015 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:27,015 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:27,015 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:27,015 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:27,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:27,180 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 01:56:27,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832647180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832647181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832647182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832647193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832647193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832647283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832647285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,428 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407c46166f218944894a54d51ec7efd74de to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c46166f218944894a54d51ec7efd74de 2023-04-07 01:56:27,432 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/3e6e6ce3c1b840e38eaab8f27dbfe71b 2023-04-07 01:56:27,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832647486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832647487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832647685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832647695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832647699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832647790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832647791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,840 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/c56c640e5c2a41eb926559a5ca8a84a8 2023-04-07 01:56:27,849 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/9c11728ba84a4ef8be4c432633654097 2023-04-07 01:56:27,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/3e6e6ce3c1b840e38eaab8f27dbfe71b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b 2023-04-07 01:56:27,855 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b, entries=150, sequenceid=50, filesize=25.1 K 2023-04-07 01:56:27,855 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/c56c640e5c2a41eb926559a5ca8a84a8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8 2023-04-07 01:56:27,858 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8, entries=150, sequenceid=50, filesize=11.6 K 2023-04-07 01:56:27,859 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/9c11728ba84a4ef8be4c432633654097 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097 2023-04-07 01:56:27,862 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097, entries=150, sequenceid=50, filesize=11.6 K 2023-04-07 01:56:27,862 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 848ms, sequenceid=50, compaction requested=true 2023-04-07 01:56:27,862 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:27,862 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:27,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:27,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:27,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:27,862 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:27,870 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:27,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:27,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0f9cc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:27,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:27,872 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,873 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:27,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:27,873 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:27,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,873 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:27,874 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:27,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:27,878 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:27,878 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:27,878 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:27,878 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:27,878 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:27,879 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:27,880 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:27,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:27,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:27,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:27,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:27,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:27,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:27,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:27,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:27,884 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:27,885 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:27,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:27,985 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:27,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:27,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:27,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:27,988 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:27,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:27,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:27,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:27,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,990 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:27,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:27,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:27,990 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:27,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:27,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:27,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:27,991 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:27,992 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:27,992 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:28,004 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040713ba29c2308048c7b381930fe6927bcf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040713ba29c2308048c7b381930fe6927bcf 2023-04-07 01:56:28,008 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/ba3a6f8808cd434dbfbe8db9cc8a4abe 2023-04-07 01:56:28,017 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/2082445c9e2b4e8b8b3af8b69ef898be 2023-04-07 01:56:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:28,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:28,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832648308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832648309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832648411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832648412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,425 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/5422085766784ca6806a6b830d215c0e 2023-04-07 01:56:28,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/ba3a6f8808cd434dbfbe8db9cc8a4abe as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe 2023-04-07 01:56:28,431 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe, entries=150, sequenceid=68, filesize=25.1 K 2023-04-07 01:56:28,432 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/2082445c9e2b4e8b8b3af8b69ef898be as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be 2023-04-07 01:56:28,434 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be, entries=150, sequenceid=68, filesize=11.6 K 2023-04-07 01:56:28,435 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/5422085766784ca6806a6b830d215c0e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e 2023-04-07 01:56:28,437 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e, entries=150, sequenceid=68, filesize=11.6 K 2023-04-07 01:56:28,438 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 447ms, sequenceid=68, compaction requested=true 2023-04-07 01:56:28,438 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:28,438 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:28,440 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41663228[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:28,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:28,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:28,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:28,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:28,448 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:28,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:28,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:28,448 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:28,448 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:28,449 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:28,450 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,451 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,453 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:28,454 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:28,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:28,454 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,454 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:28,455 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:28,555 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:28,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:28,558 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:28,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:28,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:28,560 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,560 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:28,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:28,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:28,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:28,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:28,561 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:28,562 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:28,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:28,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:28,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832648635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832648637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832648690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832648704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832648708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832648738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832648740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832648941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832648943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:28,974 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304072cabab91894c46539335adc513a385f8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072cabab91894c46539335adc513a385f8 2023-04-07 01:56:28,979 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/771642c4dcfe43639c2e5aaf7ccd5979 2023-04-07 01:56:28,987 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/f56df9b05f36407693800fc0bc105fc1 2023-04-07 01:56:29,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832649243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832649244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,396 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/21a4797f6c02436aa55ee7f86125c8a2 2023-04-07 01:56:29,402 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/771642c4dcfe43639c2e5aaf7ccd5979 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979 2023-04-07 01:56:29,406 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979, entries=150, sequenceid=82, filesize=25.1 K 2023-04-07 01:56:29,407 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/f56df9b05f36407693800fc0bc105fc1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1 2023-04-07 01:56:29,417 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1, entries=150, sequenceid=82, filesize=11.6 K 2023-04-07 01:56:29,418 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/21a4797f6c02436aa55ee7f86125c8a2 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2 2023-04-07 01:56:29,422 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2, entries=150, sequenceid=82, filesize=11.6 K 2023-04-07 01:56:29,423 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 862ms, sequenceid=82, compaction requested=true 2023-04-07 01:56:29,423 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:29,423 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:29,425 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b296818[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:29,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:29,428 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,429 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,430 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:29,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:29,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:29,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:29,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:29,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:29,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:29,441 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,441 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:29,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,441 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:29,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:29,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:29,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:29,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:29,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,447 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:29,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:29,448 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:29,448 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:29,548 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:29,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:29,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:29,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:29,551 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:29,552 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:29,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:29,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:29,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:29,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:29,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:29,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:29,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:29,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:29,556 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:29,556 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:29,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:29,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:29,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832649765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832649769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832649870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:29,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832649874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:29,969 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407ab214f2dd63449e0ab11cb6d8af9a048 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ab214f2dd63449e0ab11cb6d8af9a048 2023-04-07 01:56:30,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832650075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832650078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,374 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/e110ac17c0964cfd8b5ce2daba535ecf 2023-04-07 01:56:30,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832650380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832650383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832650708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,711 DEBUG [Thread-4113] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:30,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832650716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,720 DEBUG [Thread-4115] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:30,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832650724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,728 DEBUG [Thread-4109] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:30,783 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/f0dfd88fbbef41e391f90024b4d11bf4 2023-04-07 01:56:30,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832650883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:30,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832650886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,192 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/249ee2e39569471b818b39cca629c2ec 2023-04-07 01:56:31,195 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/e110ac17c0964cfd8b5ce2daba535ecf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf 2023-04-07 01:56:31,198 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf, entries=150, sequenceid=100, filesize=25.1 K 2023-04-07 01:56:31,199 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/f0dfd88fbbef41e391f90024b4d11bf4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4 2023-04-07 01:56:31,202 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4, entries=150, sequenceid=100, filesize=11.6 K 2023-04-07 01:56:31,203 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/249ee2e39569471b818b39cca629c2ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec 2023-04-07 01:56:31,205 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec, entries=150, sequenceid=100, filesize=11.6 K 2023-04-07 01:56:31,206 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1650ms, sequenceid=100, compaction requested=true 2023-04-07 01:56:31,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:31,206 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:31,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:31,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:31,206 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:31,208 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,208 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6d37b9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:31,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:31,210 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,216 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:31,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:31,218 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:31,221 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:31,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:31,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:31,221 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:31,221 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:31,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,223 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,224 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:31,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:31,227 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,227 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:31,227 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:31,227 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:31,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,328 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:31,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:31,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:31,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:31,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:31,331 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:31,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:31,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:31,331 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:31,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,332 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,333 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:31,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:31,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,333 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:31,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:31,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:31,335 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:31,336 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,363 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040739472f17235d4717b0405306267f6225 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040739472f17235d4717b0405306267f6225 2023-04-07 01:56:31,367 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/17d49993c7674ec3922b2c267d397c8d 2023-04-07 01:56:31,775 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/32f89ee555c74f72a313349ffe14d232 2023-04-07 01:56:31,784 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/911235177c274f59b1bc60a59633fe7a 2023-04-07 01:56:31,787 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/17d49993c7674ec3922b2c267d397c8d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d 2023-04-07 01:56:31,791 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d, entries=150, sequenceid=114, filesize=25.1 K 2023-04-07 01:56:31,792 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/32f89ee555c74f72a313349ffe14d232 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232 2023-04-07 01:56:31,796 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232, entries=150, sequenceid=114, filesize=11.6 K 2023-04-07 01:56:31,796 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/911235177c274f59b1bc60a59633fe7a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a 2023-04-07 01:56:31,799 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a, entries=150, sequenceid=114, filesize=11.6 K 2023-04-07 01:56:31,800 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 465ms, sequenceid=114, compaction requested=true 2023-04-07 01:56:31,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:31,800 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:31,802 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,802 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:31,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d84dba1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:31,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:31,804 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:31,811 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,813 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:31,816 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:31,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:31,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:31,816 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:31,816 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:31,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,818 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,821 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:31,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:31,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:31,824 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:31,824 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,825 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:31,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,925 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:31,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:31,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:31,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:31,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:31,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:31,932 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:31,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:31,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:31,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:31,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,936 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,937 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:31,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:31,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,937 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:31,940 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:31,940 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:31,940 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:31,940 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:31,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d83554b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:31,945 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:31,945 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,947 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:31,947 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,947 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:31,951 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:31,955 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:31,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:31,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:31,955 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:31,955 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:31,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,958 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,960 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:31,963 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:31,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:31,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:31,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:31,964 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:31,963 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:31,964 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:31,964 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:31,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832591948, max timestamp=1680832591977Num uniques 859; 2023-04-07 01:56:31,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:31,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832591948, max timestamp=1680832591977Num uniques 859; 2023-04-07 01:56:31,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832591948, max timestamp=1680832591977Num uniques 859; 2023-04-07 01:56:31,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:31,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:32,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832652024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832652028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,064 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,068 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,068 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,074 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2a814b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,076 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,080 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,080 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,080 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,085 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,085 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,085 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,085 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,087 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,089 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,089 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,089 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,089 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832652130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832652133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,190 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,204 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,204 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,205 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,206 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,206 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,207 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,207 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,209 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,209 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,210 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,210 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,211 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,211 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,213 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e49a3a8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,213 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,214 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,214 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,215 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,216 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:32,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,219 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,219 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,220 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,222 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,223 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,224 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,225 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,225 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,225 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,225 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,325 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832652334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832652337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,341 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,342 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,343 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,343 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,344 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,344 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407eece8167f0e543fcb99abf16f680f86a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eece8167f0e543fcb99abf16f680f86a 2023-04-07 01:56:32,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,352 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,356 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,356 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256b3213[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,374 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,374 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,375 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,374 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,375 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:32,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,378 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,378 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,378 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,378 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,379 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,381 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,386 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,390 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,391 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,391 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,491 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,494 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,494 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,495 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,495 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,496 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,496 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,498 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,498 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,498 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,498 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,501 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f502f8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,502 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,504 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,504 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,505 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,507 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,507 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,507 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,510 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,515 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,515 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:32,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,616 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,619 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,620 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,622 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,624 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,624 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,629 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2becf934[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,630 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,631 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,636 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,636 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,636 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,637 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:32,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,644 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,644 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832652644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:32,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832652644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,744 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,748 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,749 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,750 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:32,753 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,755 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:32,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382b4b02[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:32,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:32,759 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:32,765 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,765 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:32,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:32,766 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:32,766 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:32,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:32,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,773 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:32,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:32,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:32,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,773 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:32,773 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:32,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,781 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/6953146e6ca3417aa8f3cc2a3494ec07 2023-04-07 01:56:32,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/d10ba15b310a44fa8b70d9f156f57578 2023-04-07 01:56:32,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/4b1be77d852f47929b70573a0d2b3891 2023-04-07 01:56:32,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/6953146e6ca3417aa8f3cc2a3494ec07 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07 2023-04-07 01:56:32,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07, entries=200, sequenceid=124, filesize=31.9 K 2023-04-07 01:56:32,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/d10ba15b310a44fa8b70d9f156f57578 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578 2023-04-07 01:56:32,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578, entries=150, sequenceid=124, filesize=11.6 K 2023-04-07 01:56:32,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/4b1be77d852f47929b70573a0d2b3891 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891 2023-04-07 01:56:32,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891, entries=150, sequenceid=124, filesize=11.6 K 2023-04-07 01:56:32,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 907ms, sequenceid=124, compaction requested=true 2023-04-07 01:56:32,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:32,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:32,834 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-07 01:56:32,834 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-07 01:56:32,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:32,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 212908 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95088 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:32,843 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,843 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,843 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=207.9 K 2023-04-07 01:56:32,843 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=92.9 K 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting f56f0a93b92c42de8ab7cc7f59e115ec, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832585917 2023-04-07 01:56:32,843 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 606615652abd41f4868e6229f4b3d2bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832585917 2023-04-07 01:56:32,844 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting adfe3af564324f0184f749376411e1e5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832585933 2023-04-07 01:56:32,844 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0bb79ea0e34f47ba9cfc137ba4bff7c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832585933 2023-04-07 01:56:32,844 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3e6e6ce3c1b840e38eaab8f27dbfe71b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680832586550 2023-04-07 01:56:32,845 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c56c640e5c2a41eb926559a5ca8a84a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680832586550 2023-04-07 01:56:32,845 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting ba3a6f8808cd434dbfbe8db9cc8a4abe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680832587176 2023-04-07 01:56:32,845 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2082445c9e2b4e8b8b3af8b69ef898be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680832587176 2023-04-07 01:56:32,845 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 771642c4dcfe43639c2e5aaf7ccd5979, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680832588300 2023-04-07 01:56:32,845 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting e110ac17c0964cfd8b5ce2daba535ecf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832588623 2023-04-07 01:56:32,846 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 17d49993c7674ec3922b2c267d397c8d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680832589758 2023-04-07 01:56:32,846 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f56df9b05f36407693800fc0bc105fc1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680832588300 2023-04-07 01:56:32,847 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 6953146e6ca3417aa8f3cc2a3494ec07, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:32,847 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting f0dfd88fbbef41e391f90024b4d11bf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832588623 2023-04-07 01:56:32,847 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 32f89ee555c74f72a313349ffe14d232, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680832589758 2023-04-07 01:56:32,848 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d10ba15b310a44fa8b70d9f156f57578, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:32,865 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#585 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:32,869 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#584 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:32,873 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:32,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:32,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:32,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:32,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:32,882 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:32,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:32,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:32,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,884 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,884 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:32,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:32,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:32,885 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:32,885 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:32,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,885 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/276997731aca48f2a119890a4359a04e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/276997731aca48f2a119890a4359a04e 2023-04-07 01:56:32,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:32,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:32,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:32,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:32,887 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:32,887 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:32,891 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into 276997731aca48f2a119890a4359a04e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:32,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:32,891 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=8, startTime=1680832592834; duration=0sec 2023-04-07 01:56:32,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:32,891 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-07 01:56:32,893 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95088 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-07 01:56:32,893 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:32,893 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:32,893 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=92.9 K 2023-04-07 01:56:32,894 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a1c5eec9e52442fab626a196a525523e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680832585917 2023-04-07 01:56:32,894 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting cb9c76d99b0d4361ad4199aa0b95c1ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680832585933 2023-04-07 01:56:32,894 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9c11728ba84a4ef8be4c432633654097, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680832586550 2023-04-07 01:56:32,895 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5422085766784ca6806a6b830d215c0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680832587176 2023-04-07 01:56:32,895 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 21a4797f6c02436aa55ee7f86125c8a2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680832588300 2023-04-07 01:56:32,895 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 249ee2e39569471b818b39cca629c2ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680832588623 2023-04-07 01:56:32,895 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 911235177c274f59b1bc60a59633fe7a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680832589758 2023-04-07 01:56:32,896 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 4b1be77d852f47929b70573a0d2b3891, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:32,906 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304078c92d015cfa84e52b87611cd32766b7d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078c92d015cfa84e52b87611cd32766b7d 2023-04-07 01:56:32,911 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#587 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:32,924 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/76ebd0a9e359432c8d517e067e216243 2023-04-07 01:56:32,929 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/dc2c684b900a40f0aa3ca541580f3254 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/dc2c684b900a40f0aa3ca541580f3254 2023-04-07 01:56:32,933 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into dc2c684b900a40f0aa3ca541580f3254(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:32,933 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:32,933 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=8, startTime=1680832592834; duration=0sec 2023-04-07 01:56:32,933 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:32,933 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/9b9dea9227fe4d01b0c1ecbfa315457b 2023-04-07 01:56:33,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:33,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:33,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832653162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832653163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832653266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40144 deadline: 1680832653268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,325 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/9ba3dee1a9d742f4b1a3b56d0ed19117 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/9ba3dee1a9d742f4b1a3b56d0ed19117 2023-04-07 01:56:33,330 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into 9ba3dee1a9d742f4b1a3b56d0ed19117(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:33,330 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:33,330 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=8, startTime=1680832592834; duration=0sec 2023-04-07 01:56:33,330 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:33,344 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/c744075e15444fd991d0c20225a5b1a7 2023-04-07 01:56:33,347 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/76ebd0a9e359432c8d517e067e216243 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243 2023-04-07 01:56:33,350 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243, entries=150, sequenceid=161, filesize=25.3 K 2023-04-07 01:56:33,351 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/9b9dea9227fe4d01b0c1ecbfa315457b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b 2023-04-07 01:56:33,353 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b, entries=150, sequenceid=161, filesize=11.8 K 2023-04-07 01:56:33,354 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/c744075e15444fd991d0c20225a5b1a7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7 2023-04-07 01:56:33,356 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7, entries=150, sequenceid=161, filesize=11.8 K 2023-04-07 01:56:33,357 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 470ms, sequenceid=161, compaction requested=false 2023-04-07 01:56:33,357 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:33,357 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:33,359 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,360 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c0dbe[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:33,361 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:33,361 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,363 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:33,363 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,363 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:33,365 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:33,367 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:33,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:33,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:33,367 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:33,367 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:33,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,370 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,371 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,372 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:33,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:33,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,376 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,376 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,377 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:33,377 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:33,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,377 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:33,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:33,477 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:33,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:33,484 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:33,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:33,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:33,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,484 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,486 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,486 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,486 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,486 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:33,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:33,487 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304072e9264cd25694b26ac0faf658d480611 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e9264cd25694b26ac0faf658d480611 2023-04-07 01:56:33,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:33,489 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,489 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:33,489 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:33,489 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:33,494 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=175, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/bc3d365d4b7f4f6e959c64dde09f8ecf 2023-04-07 01:56:33,494 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,496 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:33,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:33,497 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e742501[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:33,498 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:33,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:33,498 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:33,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,499 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:33,501 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:33,502 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,502 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:33,502 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:33,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:33,506 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:33,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:33,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:33,506 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:33,506 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:33,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,509 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,509 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832653508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832593494, max timestamp=1680832593509Num uniques 859; 2023-04-07 01:56:33,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:33,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832593494, max timestamp=1680832593509Num uniques 859; 2023-04-07 01:56:33,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:33,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832593494, max timestamp=1680832593509Num uniques 859; 2023-04-07 01:56:33,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,515 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:33,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:33,516 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:33,516 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:33,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:33,515 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832653566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40064 deadline: 1680832653612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,616 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:33,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:33,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:33,619 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,619 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,621 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:33,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:33,622 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:33,622 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:33,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:33,623 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,625 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:33,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321d46a2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:33,627 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,628 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:33,630 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:33,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,630 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:33,632 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:33,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:33,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:33,632 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:33,632 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:33,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,634 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,635 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,637 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,643 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:33,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:33,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:33,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,643 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:33,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,643 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,644 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:33,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832653669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,744 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:33,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:33,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:33,747 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:33,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:33,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:33,747 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,748 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,755 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,755 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,755 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,755 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:33,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:33,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:33,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:33,757 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:33,758 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,758 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:33,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508c78b8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:33,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:33,760 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,761 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,762 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:33,762 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:33,762 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:33,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:33,763 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:33,765 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:33,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:33,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:33,766 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:33,766 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:33,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,768 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,769 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,770 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,772 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:33,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:33,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,772 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,773 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:33,773 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:33,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,773 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832653815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,873 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:33,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832653872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:33,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:33,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:33,880 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:33,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:33,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:33,880 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,881 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,882 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,882 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,883 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:33,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:33,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:33,883 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:33,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:33,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:33,884 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:33,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,889 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,889 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e9ceea[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:33,891 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:33,891 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,893 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:33,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:33,893 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:33,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:33,896 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:33,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:33,899 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:33,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:33,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:33,899 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:33,899 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:33,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:33,903 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/0d2096f228ae45918476722115e320e6 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:33,908 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:33,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:33,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:33,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:33,908 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:33,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:33,909 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:33,909 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:33,909 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:33,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:33,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:33,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/7516ab31423e4b8c809e584e1c066c80 2023-04-07 01:56:33,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/bc3d365d4b7f4f6e959c64dde09f8ecf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf 2023-04-07 01:56:33,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf, entries=200, sequenceid=175, filesize=32.1 K 2023-04-07 01:56:33,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/0d2096f228ae45918476722115e320e6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6 2023-04-07 01:56:33,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6, entries=150, sequenceid=175, filesize=11.8 K 2023-04-07 01:56:33,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/7516ab31423e4b8c809e584e1c066c80 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80 2023-04-07 01:56:33,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80, entries=150, sequenceid=175, filesize=11.8 K 2023-04-07 01:56:33,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 484ms, sequenceid=175, compaction requested=true 2023-04-07 01:56:33,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:33,957 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:33,957 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:33,957 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:33,958 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,958 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:33,958 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/9ba3dee1a9d742f4b1a3b56d0ed19117, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=82.8 K 2023-04-07 01:56:33,958 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/276997731aca48f2a119890a4359a04e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=35.4 K 2023-04-07 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 9ba3dee1a9d742f4b1a3b56d0ed19117, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 276997731aca48f2a119890a4359a04e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 76ebd0a9e359432c8d517e067e216243, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680832592007 2023-04-07 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9b9dea9227fe4d01b0c1ecbfa315457b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680832592007 2023-04-07 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting bc3d365d4b7f4f6e959c64dde09f8ecf, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 0d2096f228ae45918476722115e320e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:33,967 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#593 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:33,975 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#594 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:34,009 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:34,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:34,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:34,012 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:34,012 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:34,012 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:34,012 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:34,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:34,013 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:34,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,014 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:34,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:34,014 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,015 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:34,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:34,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:34,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:34,016 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:34,016 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:34,016 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:34,016 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:34,016 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:34,016 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:34,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:34,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:34,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832654147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832654175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832654251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,381 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/7a1d1a8f9329481b9485925bca5274c4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/7a1d1a8f9329481b9485925bca5274c4 2023-04-07 01:56:34,384 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into 7a1d1a8f9329481b9485925bca5274c4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:34,384 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:34,384 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=13, startTime=1680832593957; duration=0sec 2023-04-07 01:56:34,384 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:34,384 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:34,385 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:34,385 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:34,385 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:34,385 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/dc2c684b900a40f0aa3ca541580f3254, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=35.4 K 2023-04-07 01:56:34,385 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dc2c684b900a40f0aa3ca541580f3254, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680832591916 2023-04-07 01:56:34,386 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c744075e15444fd991d0c20225a5b1a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680832592007 2023-04-07 01:56:34,386 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7516ab31423e4b8c809e584e1c066c80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:34,391 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#596 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:34,398 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/a6b89dd6624942b498af76e9bf5878fc as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a6b89dd6624942b498af76e9bf5878fc 2023-04-07 01:56:34,401 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into a6b89dd6624942b498af76e9bf5878fc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:34,402 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:34,402 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=13, startTime=1680832593957; duration=0sec 2023-04-07 01:56:34,402 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:34,427 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407cc031af133c34f90adbccca560151562 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cc031af133c34f90adbccca560151562 2023-04-07 01:56:34,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832654457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832654680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832654734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,736 DEBUG [Thread-4113] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832654746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,750 DEBUG [Thread-4109] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:34,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832654748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,752 DEBUG [Thread-4115] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:34,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:34,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832654760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:34,802 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7f03a20be2644a808c5c2078d83aac3b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7f03a20be2644a808c5c2078d83aac3b 2023-04-07 01:56:34,806 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into 7f03a20be2644a808c5c2078d83aac3b(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:34,806 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:34,806 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=13, startTime=1680832593957; duration=0sec 2023-04-07 01:56:34,806 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:34,833 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/1b2dafbb15b9490795fffdabd1d11a98 2023-04-07 01:56:35,261 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/39840401830b44fbbd64cf105f42bcbb 2023-04-07 01:56:35,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:35,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832655266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,671 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/685c254da2cd4cc7b1eb4463ffe0bd1a 2023-04-07 01:56:35,677 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/1b2dafbb15b9490795fffdabd1d11a98 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98 2023-04-07 01:56:35,683 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98, entries=150, sequenceid=211, filesize=25.3 K 2023-04-07 01:56:35,684 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/39840401830b44fbbd64cf105f42bcbb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb 2023-04-07 01:56:35,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832655687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,693 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb, entries=150, sequenceid=211, filesize=11.8 K 2023-04-07 01:56:35,694 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/685c254da2cd4cc7b1eb4463ffe0bd1a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a 2023-04-07 01:56:35,698 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a, entries=150, sequenceid=211, filesize=11.8 K 2023-04-07 01:56:35,698 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1682ms, sequenceid=211, compaction requested=false 2023-04-07 01:56:35,699 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:35,699 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:35,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:35,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:35,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:35,699 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:35,700 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,700 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:35,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2beab15b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:35,703 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:35,703 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,704 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,704 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:35,704 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:35,704 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:35,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,705 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:35,707 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:35,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,707 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,707 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:35,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:35,711 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:35,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:35,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:35,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:35,711 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:35,712 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,712 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,712 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:35,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:35,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:35,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,713 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,719 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:35,719 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:35,719 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:35,719 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:35,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:35,719 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,720 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,720 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,720 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,720 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,720 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:35,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:35,720 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:35,720 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:35,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:35,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:35,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:35,820 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:35,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:35,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:35,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:35,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:35,824 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:35,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:35,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:35,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:35,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,824 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:35,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:35,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:35,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:35,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:35,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:35,828 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:35,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:35,832 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:35,832 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:35,832 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:35,832 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:35,832 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:35,832 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:35,833 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:35,833 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:35,833 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:35,833 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:35,847 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040766a756b0ad8944a394cd42a9f639fd22 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040766a756b0ad8944a394cd42a9f639fd22 2023-04-07 01:56:35,864 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=225, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/4caf3cc070194696af9e26ec9848ea01 2023-04-07 01:56:35,878 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/80ea3de5dd83466cbcffd210bc5b218b 2023-04-07 01:56:35,904 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/2450d32609314c5fb5e557cd13da5e8e 2023-04-07 01:56:35,908 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/4caf3cc070194696af9e26ec9848ea01 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01 2023-04-07 01:56:35,913 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01, entries=150, sequenceid=225, filesize=25.3 K 2023-04-07 01:56:35,913 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/80ea3de5dd83466cbcffd210bc5b218b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b 2023-04-07 01:56:35,917 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b, entries=150, sequenceid=225, filesize=11.8 K 2023-04-07 01:56:35,917 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/2450d32609314c5fb5e557cd13da5e8e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e 2023-04-07 01:56:35,920 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e, entries=150, sequenceid=225, filesize=11.8 K 2023-04-07 01:56:35,921 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 89ms, sequenceid=225, compaction requested=true 2023-04-07 01:56:35,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:35,921 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:35,928 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,928 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:35,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:35,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739acad2[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:35,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:35,931 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,932 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:35,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,933 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:35,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:35,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:35,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:35,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:35,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:35,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:35,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:35,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:35,939 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:35,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:35,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:35,944 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:35,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:35,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:35,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:35,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:35,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:35,945 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:35,945 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:35,945 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:35,945 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:35,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:35,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:35,945 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,045 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:36,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:36,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:36,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:36,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:36,049 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:36,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:36,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:36,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,049 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,051 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:36,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:36,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,051 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:36,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:36,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:36,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:36,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:36,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:36,053 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:36,055 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,055 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:36,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e035a1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:36,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:36,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,062 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:36,062 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:36,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,064 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:36,064 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:36,064 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:36,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:36,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:36,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:36,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:36,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:36,067 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:36,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:36,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:36,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:36,073 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:36,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:36,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:36,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:36,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:36,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:36,074 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:36,074 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:36,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:36,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:36,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:36,174 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:36,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:36,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:36,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:36,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:36,178 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:36,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:36,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:36,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,178 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:36,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:36,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:36,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:36,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:36,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:36,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:36,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:36,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,184 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:36,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a443dc9[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:36,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:36,186 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,187 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:36,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:36,187 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:36,187 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,187 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:36,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:36,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:36,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:36,191 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:36,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:36,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:36,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:36,192 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:36,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,195 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,195 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:36,197 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:36,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:36,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:36,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:36,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,200 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:36,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:36,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:36,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:36,200 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,200 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:36,200 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:36,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:36,300 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:36,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:36,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:36,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:36,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:36,304 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:36,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:36,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:36,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,305 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,306 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:36,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,309 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:36,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:36,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:36,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:36,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:36,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:36,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:36,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:36,311 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 01:56:36,311 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:36,312 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:36,312 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:36,312 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:36,335 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:36,335 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:36,368 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e48baa7ec70e4ed28fa37ba59911bbff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e48baa7ec70e4ed28fa37ba59911bbff 2023-04-07 01:56:36,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:36,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832596386, max timestamp=1680832596410Num uniques 859; 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832596386, max timestamp=1680832596410Num uniques 859; 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832596386, max timestamp=1680832596410Num uniques 859; 2023-04-07 01:56:36,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832596453, max timestamp=1680832596467Num uniques 1018; 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832596453, max timestamp=1680832596467Num uniques 1018; 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680832596453, max timestamp=1680832596467Num uniques 1018; 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 01:56:36,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832596492, max timestamp=1680832596515Num uniques 1027; 2023-04-07 01:56:36,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:56:36,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832596492, max timestamp=1680832596515Num uniques 1027; 2023-04-07 01:56:36,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680832596492, max timestamp=1680832596515Num uniques 1027; 2023-04-07 01:56:36,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:56:36,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 01:56:36,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680832596554, max timestamp=1680832596571Num uniques 1036; 2023-04-07 01:56:36,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 01:56:36,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680832596554, max timestamp=1680832596571Num uniques 1036; 2023-04-07 01:56:36,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 01:56:36,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680832596554, max timestamp=1680832596571Num uniques 1036; 2023-04-07 01:56:36,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832596611, max timestamp=1680832596622Num uniques 1095; 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680832596611, max timestamp=1680832596622Num uniques 1095; 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680832596614, max timestamp=1680832596622Num uniques 1095; 2023-04-07 01:56:36,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-07 01:56:36,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832596671, max timestamp=1680832596682Num uniques 1104; 2023-04-07 01:56:36,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-07 01:56:36,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832596671, max timestamp=1680832596682Num uniques 1054; 2023-04-07 01:56:36,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-07 01:56:36,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680832596671, max timestamp=1680832596682Num uniques 1104; 2023-04-07 01:56:36,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832596715, max timestamp=1680832596732Num uniques 1013; 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832596715, max timestamp=1680832596732Num uniques 1013; 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680832596715, max timestamp=1680832596732Num uniques 1013; 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 01:56:36,778 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/3a22193a17c5447ea9c53def85829efa 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832596771, max timestamp=1680832596792Num uniques 1022; 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832596771, max timestamp=1680832596792Num uniques 1022; 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680832596771, max timestamp=1680832596792Num uniques 1022; 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:56:36,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832596839, max timestamp=1680832596864Num uniques 1031; 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832596839, max timestamp=1680832596864Num uniques 1031; 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680832596839, max timestamp=1680832596864Num uniques 1031; 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:56:36,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-07 01:56:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832596880, max timestamp=1680832596922Num uniques 1040; 2023-04-07 01:56:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:56:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832596880, max timestamp=1680832596922Num uniques 1040; 2023-04-07 01:56:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:56:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680832596880, max timestamp=1680832596922Num uniques 1040; 2023-04-07 01:56:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-07 01:56:36,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832596930, max timestamp=1680832596968Num uniques 1049; 2023-04-07 01:56:36,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:56:36,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832596930, max timestamp=1680832596968Num uniques 1049; 2023-04-07 01:56:36,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:56:36,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680832596930, max timestamp=1680832596968Num uniques 1049; 2023-04-07 01:56:36,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832597013, max timestamp=1680832597029Num uniques 1058; 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832597013, max timestamp=1680832597029Num uniques 1058; 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680832597013, max timestamp=1680832597029Num uniques 1058; 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:56:37,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-07 01:56:37,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680832597074, max timestamp=1680832597086Num uniques 1017; 2023-04-07 01:56:37,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-07 01:56:37,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680832597074, max timestamp=1680832597086Num uniques 1017; 2023-04-07 01:56:37,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680832597074, max timestamp=1680832597086Num uniques 1017; 2023-04-07 01:56:37,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-07 01:56:37,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832597121, max timestamp=1680832597130Num uniques 1026; 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832597121, max timestamp=1680832597130Num uniques 1026; 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680832597121, max timestamp=1680832597130Num uniques 1026; 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-07 01:56:37,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-07 01:56:37,189 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/eab9149636b741789bff7aa752784922 2023-04-07 01:56:37,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680832597166, max timestamp=1680832597189Num uniques 1035; 2023-04-07 01:56:37,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680832597166, max timestamp=1680832597189Num uniques 1035; 2023-04-07 01:56:37,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-07 01:56:37,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-07 01:56:37,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680832597166, max timestamp=1680832597189Num uniques 1035; 2023-04-07 01:56:37,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=6870, suffix heap size=135608,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=150 2023-04-07 01:56:37,242 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/8387993ba4cb46e0adfb9f828e273ebd 2023-04-07 01:56:37,251 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/3a22193a17c5447ea9c53def85829efa as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa 2023-04-07 01:56:37,255 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa, entries=50, sequenceid=231, filesize=11.6 K 2023-04-07 01:56:37,256 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/eab9149636b741789bff7aa752784922 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922 2023-04-07 01:56:37,260 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922, entries=50, sequenceid=231, filesize=7.1 K 2023-04-07 01:56:37,260 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/8387993ba4cb46e0adfb9f828e273ebd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd 2023-04-07 01:56:37,264 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd, entries=50, sequenceid=231, filesize=7.1 K 2023-04-07 01:56:37,265 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=99.07 KB/101450 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 954ms, sequenceid=231, compaction requested=true 2023-04-07 01:56:37,265 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:37,265 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:37,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:37,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=105.78 KB heapSize=299.30 KB 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:37,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,268 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a3bb3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:37,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:37,271 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:37,275 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:37,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:37,281 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:37,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:37,281 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:37,282 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:37,282 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:37,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,283 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,284 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,287 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,290 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,290 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,292 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,292 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:37,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:37,296 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:37,296 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:37,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:37,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832657316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,323 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407c1c3a94422664f4fa7f5a5232f6aa043 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c1c3a94422664f4fa7f5a5232f6aa043 2023-04-07 01:56:37,396 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:37,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:37,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:37,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:37,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:37,399 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:37,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:37,399 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:37,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,402 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:37,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:37,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:37,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:37,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,403 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:37,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:37,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2133adb0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:37,407 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:37,407 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,408 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:37,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:37,408 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:37,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:37,413 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:37,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,413 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:37,415 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:37,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:37,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:37,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:37,415 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:37,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,417 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,419 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:37,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832657418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,421 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:37,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:37,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,422 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,422 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:37,422 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:37,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,522 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:37,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:37,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:37,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:37,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:37,525 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:37,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:37,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:37,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,527 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,529 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,529 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:37,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:37,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:37,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:37,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:37,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:37,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:37,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:37,530 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:37,531 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,531 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:37,531 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:37,531 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:37,533 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:37,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603fc90a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:37,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:37,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,539 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,539 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:37,539 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:37,539 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:37,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,539 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:37,541 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:37,541 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,541 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,541 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:37,544 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:37,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:37,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:37,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:37,544 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:37,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,546 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,547 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,550 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,550 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:37,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:37,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,550 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:37,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,551 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:37,551 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,551 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:37,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832657622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,651 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:37,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:37,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:37,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:37,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:37,658 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:37,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:37,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:37,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,659 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,660 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,660 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,661 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:37,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:37,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:37,662 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,662 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:37,662 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:37,662 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:37,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:37,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:37,663 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:37,664 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bce0ab1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:37,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:37,666 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,667 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:37,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:37,667 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:37,669 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,669 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:37,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,670 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:37,672 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:37,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:37,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:37,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:37,672 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:37,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,674 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:37,674 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:37,675 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,678 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:37,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:37,678 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:37,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:37,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:37,678 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:37,679 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:37,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832657708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,711 DEBUG [Thread-4111] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:37,735 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=504, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/0ced0ea3aca847bcba1dbd4d95757e5a 2023-04-07 01:56:37,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/525ca8e1adad4fb7b81efa4ff6b1cbe7 2023-04-07 01:56:37,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=34.22 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/09fef5c2ce324ded93cdfac8e948f63a 2023-04-07 01:56:37,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/0ced0ea3aca847bcba1dbd4d95757e5a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a 2023-04-07 01:56:37,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a, entries=150, sequenceid=504, filesize=25.4 K 2023-04-07 01:56:37,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/525ca8e1adad4fb7b81efa4ff6b1cbe7 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7 2023-04-07 01:56:37,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7, entries=150, sequenceid=504, filesize=11.9 K 2023-04-07 01:56:37,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/09fef5c2ce324ded93cdfac8e948f63a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a 2023-04-07 01:56:37,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a, entries=150, sequenceid=504, filesize=11.9 K 2023-04-07 01:56:37,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~105.78 KB/108320, heapSize ~299.26 KB/306440, currentSize=73.80 KB/75570 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 501ms, sequenceid=504, compaction requested=true 2023-04-07 01:56:37,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:37,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:37,768 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:56:37,768 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:56:37,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:37,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:37,769 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 116016 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:56:37,769 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55920 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:56:37,769 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:37,770 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:37,770 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,770 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,770 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7f03a20be2644a808c5c2078d83aac3b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=113.3 K 2023-04-07 01:56:37,770 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/7a1d1a8f9329481b9485925bca5274c4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=54.6 K 2023-04-07 01:56:37,770 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7f03a20be2644a808c5c2078d83aac3b, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:37,770 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7a1d1a8f9329481b9485925bca5274c4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:37,770 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 1b2dafbb15b9490795fffdabd1d11a98, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680832593556 2023-04-07 01:56:37,771 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 39840401830b44fbbd64cf105f42bcbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680832593556 2023-04-07 01:56:37,771 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 4caf3cc070194696af9e26ec9848ea01, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680832594139 2023-04-07 01:56:37,771 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 80ea3de5dd83466cbcffd210bc5b218b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680832594139 2023-04-07 01:56:37,771 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 3a22193a17c5447ea9c53def85829efa, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832596294 2023-04-07 01:56:37,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting eab9149636b741789bff7aa752784922, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832596294 2023-04-07 01:56:37,772 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 0ced0ea3aca847bcba1dbd4d95757e5a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:37,772 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 525ca8e1adad4fb7b81efa4ff6b1cbe7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:37,779 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:37,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:37,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:37,784 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:37,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:37,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:37,785 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,785 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,787 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:37,787 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:37,788 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#609 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:37,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:37,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:37,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:37,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:37,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:37,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:37,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:37,793 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:37,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:37,794 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:37,794 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:37,794 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:37,795 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#608 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:37,804 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/5d04bf99404b466795bdafd30f79b1bd as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/5d04bf99404b466795bdafd30f79b1bd 2023-04-07 01:56:37,808 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into 5d04bf99404b466795bdafd30f79b1bd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:37,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:37,808 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=11, startTime=1680832597768; duration=0sec 2023-04-07 01:56:37,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:37,808 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 01:56:37,810 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55920 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 01:56:37,810 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:37,810 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:37,810 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a6b89dd6624942b498af76e9bf5878fc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=54.6 K 2023-04-07 01:56:37,810 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a6b89dd6624942b498af76e9bf5878fc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680832593158 2023-04-07 01:56:37,811 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 685c254da2cd4cc7b1eb4463ffe0bd1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680832593556 2023-04-07 01:56:37,811 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2450d32609314c5fb5e557cd13da5e8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680832594139 2023-04-07 01:56:37,811 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 8387993ba4cb46e0adfb9f828e273ebd, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680832596294 2023-04-07 01:56:37,812 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 09fef5c2ce324ded93cdfac8e948f63a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:37,818 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#611 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:37,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:37,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:37,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:37,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832657975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:38,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:38,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832658079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:38,214 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407768fe95a8aee46ada790b8048e385c2e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407768fe95a8aee46ada790b8048e385c2e 2023-04-07 01:56:38,225 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/bbc70c232c9e4a40bc96d0c2864b8ad8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bbc70c232c9e4a40bc96d0c2864b8ad8 2023-04-07 01:56:38,229 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into bbc70c232c9e4a40bc96d0c2864b8ad8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:38,229 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:38,229 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=11, startTime=1680832597768; duration=0sec 2023-04-07 01:56:38,229 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:38,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832658282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:38,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:38,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832658585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:38,616 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/47793098d3ba4954b9681e083e354632 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/47793098d3ba4954b9681e083e354632 2023-04-07 01:56:38,620 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into 47793098d3ba4954b9681e083e354632(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:38,620 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:38,620 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=11, startTime=1680832597768; duration=0sec 2023-04-07 01:56:38,620 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:38,621 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/95dbc97d91314f689c35818b79e7f9c6 2023-04-07 01:56:39,030 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/64d2da08326c4a83af24d4a08accaaad 2023-04-07 01:56:39,039 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/2a121f01701b44b597c9bf9ee52602bf 2023-04-07 01:56:39,042 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/95dbc97d91314f689c35818b79e7f9c6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6 2023-04-07 01:56:39,045 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6, entries=100, sequenceid=518, filesize=18.6 K 2023-04-07 01:56:39,045 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/64d2da08326c4a83af24d4a08accaaad as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad 2023-04-07 01:56:39,048 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad, entries=100, sequenceid=518, filesize=9.5 K 2023-04-07 01:56:39,048 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/2a121f01701b44b597c9bf9ee52602bf as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf 2023-04-07 01:56:39,051 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf, entries=100, sequenceid=518, filesize=9.5 K 2023-04-07 01:56:39,052 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1258ms, sequenceid=518, compaction requested=false 2023-04-07 01:56:39,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:39,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9c39d8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,055 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,056 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,057 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,066 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,071 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,071 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,072 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,072 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,072 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:39,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:39,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:39,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:39,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832659116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,172 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,175 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,176 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,180 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,184 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,187 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52fe4796[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,187 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,189 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,191 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,191 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,193 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,198 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,198 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,198 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,199 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,198 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832659218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,299 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,302 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,304 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,305 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c97a35a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,310 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,311 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,311 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,313 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,316 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,316 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,316 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,319 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,321 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,321 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,322 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,321 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:39,422 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832659421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,424 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,425 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,425 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,430 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,430 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7b3fb[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,432 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,433 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,433 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,437 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,439 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,442 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,445 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,445 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,446 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,446 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,505 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304078a234b481cf74845a26de4504fed731c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078a234b481cf74845a26de4504fed731c 2023-04-07 01:56:39,546 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,549 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,549 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,550 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,550 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,551 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,552 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,552 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,552 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,552 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,553 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@81a0822[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,555 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,556 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,558 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,561 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,561 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,561 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,563 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,567 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,567 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,567 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,668 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,671 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,671 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,672 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,672 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,673 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,674 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,674 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,675 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28783c40[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,686 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,689 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,690 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,690 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,690 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,690 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,691 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,692 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,693 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,696 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,696 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,696 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,696 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:39,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832659724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,796 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,799 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,800 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,801 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,801 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,802 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,803 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,803 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,803 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,803 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,805 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,805 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5904d766[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,806 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,807 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,807 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,810 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,813 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,813 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,813 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,815 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,817 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,819 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,819 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,820 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,820 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,909 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=540, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7c5c9f4feb124042b2a9b7c98130ac89 2023-04-07 01:56:39,920 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:39,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:39,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:39,923 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:39,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:39,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:39,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,923 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,927 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,927 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,928 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:39,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:39,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:39,929 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,929 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:39,929 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:39,929 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:39,931 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:39,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e6efd6[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:39,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:39,932 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,934 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:39,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:39,934 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:39,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,934 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,935 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:39,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:39,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:39,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:39,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:39,938 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:39,938 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:39,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,940 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:39,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:39,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,942 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,944 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:39,944 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:39,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:39,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:39,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:39,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:39,945 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:39,945 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:39,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,045 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,048 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,048 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,050 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,050 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,051 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:40,052 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,053 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b65b090[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,056 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,058 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,058 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,058 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,059 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,059 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,061 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,061 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,062 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,062 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,063 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,064 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,067 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,067 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,068 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,067 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,168 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,171 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,171 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,172 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,175 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,175 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,176 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:40,177 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,177 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,178 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,178 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,179 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52961180[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,180 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,182 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,182 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,182 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,183 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,183 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,185 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,185 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,185 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,188 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,188 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,189 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,191 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,191 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,192 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,192 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,192 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:40,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832660225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,292 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,295 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,295 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,296 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,297 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,297 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,298 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,298 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,299 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,299 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,299 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:40,299 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,300 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,300 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d564353[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,302 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,307 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,307 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,308 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,310 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,311 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,311 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,313 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,317 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,318 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,318 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/9f55b92999fd4893b06dee4300de57db 2023-04-07 01:56:40,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/a9087f11f7a040e8afa54b4ded2ca955 2023-04-07 01:56:40,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7c5c9f4feb124042b2a9b7c98130ac89 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89 2023-04-07 01:56:40,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89, entries=200, sequenceid=540, filesize=32.3 K 2023-04-07 01:56:40,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/9f55b92999fd4893b06dee4300de57db as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db 2023-04-07 01:56:40,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db, entries=150, sequenceid=540, filesize=11.9 K 2023-04-07 01:56:40,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/a9087f11f7a040e8afa54b4ded2ca955 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955 2023-04-07 01:56:40,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955, entries=150, sequenceid=540, filesize=11.9 K 2023-04-07 01:56:40,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1247ms, sequenceid=540, compaction requested=true 2023-04-07 01:56:40,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:40,341 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:40,341 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:40,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:40,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:40,342 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:40,342 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:40,342 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:40,342 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:40,342 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,343 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,343 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/47793098d3ba4954b9681e083e354632, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=76.8 K 2023-04-07 01:56:40,343 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/5d04bf99404b466795bdafd30f79b1bd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=33.8 K 2023-04-07 01:56:40,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 47793098d3ba4954b9681e083e354632, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:40,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 5d04bf99404b466795bdafd30f79b1bd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:40,343 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 95dbc97d91314f689c35818b79e7f9c6, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832597309 2023-04-07 01:56:40,343 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 64d2da08326c4a83af24d4a08accaaad, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832597309 2023-04-07 01:56:40,344 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7c5c9f4feb124042b2a9b7c98130ac89, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680832597966 2023-04-07 01:56:40,344 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 9f55b92999fd4893b06dee4300de57db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680832597969 2023-04-07 01:56:40,349 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#618 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:40,352 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#617 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:40,418 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,421 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,421 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,423 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,423 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,425 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,425 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,425 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,425 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:40,426 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:40,435 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407428ece079e3949f59f9bdc4cf7d51a6d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407428ece079e3949f59f9bdc4cf7d51a6d 2023-04-07 01:56:40,439 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=553, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/c42452d5ed81482c9da8a98d13135358 2023-04-07 01:56:40,447 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/e0dab1d2f37b435f9e848ecd916952ad 2023-04-07 01:56:40,455 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/6354ca2c51984b44bcbd62025e64df22 2023-04-07 01:56:40,459 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/c42452d5ed81482c9da8a98d13135358 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358 2023-04-07 01:56:40,462 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358, entries=150, sequenceid=553, filesize=25.4 K 2023-04-07 01:56:40,462 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/e0dab1d2f37b435f9e848ecd916952ad as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad 2023-04-07 01:56:40,465 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad, entries=150, sequenceid=553, filesize=11.9 K 2023-04-07 01:56:40,466 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/6354ca2c51984b44bcbd62025e64df22 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22 2023-04-07 01:56:40,469 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22, entries=150, sequenceid=553, filesize=11.9 K 2023-04-07 01:56:40,469 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 44ms, sequenceid=553, compaction requested=true 2023-04-07 01:56:40,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,470 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,471 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,472 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791a8170[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,473 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,474 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,474 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,475 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,476 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,487 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,487 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,487 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,489 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,489 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,489 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,490 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,491 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,493 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,493 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,494 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,494 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,594 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,597 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,597 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,598 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,598 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,599 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,599 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,601 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,601 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,602 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,602 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,603 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fed4ff3[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,604 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,604 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,606 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,606 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,607 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,608 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,610 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,610 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,611 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,611 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,612 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,613 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,614 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,616 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,616 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,617 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,617 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,617 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,717 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,721 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,723 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,723 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,724 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,725 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,726 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,727 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,727 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d15599f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,729 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,730 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,730 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,735 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,738 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,738 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,738 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,739 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,739 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,739 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,740 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,741 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,743 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,743 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,744 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,743 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,744 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,759 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/871798421f4c452b95ceaf4e4e8a8d80 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/871798421f4c452b95ceaf4e4e8a8d80 2023-04-07 01:56:40,763 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into 871798421f4c452b95ceaf4e4e8a8d80(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:40,763 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,763 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=13, startTime=1680832600341; duration=0sec 2023-04-07 01:56:40,763 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:40,763 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:40,764 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:40,765 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:40,765 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,765 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bbc70c232c9e4a40bc96d0c2864b8ad8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=45.7 K 2023-04-07 01:56:40,765 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting bbc70c232c9e4a40bc96d0c2864b8ad8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680832597236 2023-04-07 01:56:40,765 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 2a121f01701b44b597c9bf9ee52602bf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680832597309 2023-04-07 01:56:40,766 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting a9087f11f7a040e8afa54b4ded2ca955, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680832597969 2023-04-07 01:56:40,766 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 6354ca2c51984b44bcbd62025e64df22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832599106 2023-04-07 01:56:40,771 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/5ce4e59d32fe4a198b2e4c34bb9097f8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/5ce4e59d32fe4a198b2e4c34bb9097f8 2023-04-07 01:56:40,773 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#622 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:40,775 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into 5ce4e59d32fe4a198b2e4c34bb9097f8(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:40,775 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,775 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=13, startTime=1680832600341; duration=0sec 2023-04-07 01:56:40,775 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:40,844 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,847 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,847 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,848 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,848 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,850 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,850 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,851 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,852 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,852 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,853 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c6cfb5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,855 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,855 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,856 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,856 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,858 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,858 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,858 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,858 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,860 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,860 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,860 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,860 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,860 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,863 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,863 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,863 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,864 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,865 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,867 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,868 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,868 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,867 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,868 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,968 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:40,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:40,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:40,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:40,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:40,971 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:40,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:40,971 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:40,972 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,972 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,973 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,973 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:40,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:40,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:40,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:40,974 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:40,974 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:40,974 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:40,975 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,975 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:40,975 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:40,975 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:40,975 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:40,976 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,977 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,977 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:40,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffacd9c[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:40,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:40,978 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,981 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,981 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:40,983 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:40,983 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,983 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,983 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:40,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:40,985 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:40,985 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:40,985 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:40,985 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:40,986 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:40,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,987 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:40,988 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,989 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:40,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:40,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:40,991 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:40,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:40,992 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:40,992 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:40,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:40,992 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:40,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,112 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,116 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,116 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,119 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,119 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,121 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,121 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,121 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:41,121 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,122 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aed9f47[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,125 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,126 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,126 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,127 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,127 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,127 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,128 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,130 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,130 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,130 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,133 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,134 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,135 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,140 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,140 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,141 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,141 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,141 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,180 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/79ec0d441e7645bcb8f48199f800c3b4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/79ec0d441e7645bcb8f48199f800c3b4 2023-04-07 01:56:41,184 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into 79ec0d441e7645bcb8f48199f800c3b4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:41,184 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:41,184 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=12, startTime=1680832600342; duration=0sec 2023-04-07 01:56:41,184 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:41,241 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,244 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,244 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,245 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,246 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,246 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,248 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:41,248 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,258 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040757e9b38d940b430db63b1bf06662a083 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757e9b38d940b430db63b1bf06662a083 2023-04-07 01:56:41,263 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=566, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/8f03660eac3e4e3dbc8bdf7fc0117d41 2023-04-07 01:56:41,271 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/ad08a517343f4c75a6a1e59ad16078e5 2023-04-07 01:56:41,279 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/b6a969a8ff00473f9e27cdbf08a664f1 2023-04-07 01:56:41,283 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/8f03660eac3e4e3dbc8bdf7fc0117d41 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41 2023-04-07 01:56:41,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:41,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,286 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41, entries=100, sequenceid=566, filesize=18.6 K 2023-04-07 01:56:41,287 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/ad08a517343f4c75a6a1e59ad16078e5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5 2023-04-07 01:56:41,290 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5, entries=50, sequenceid=566, filesize=7.1 K 2023-04-07 01:56:41,290 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/b6a969a8ff00473f9e27cdbf08a664f1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1 2023-04-07 01:56:41,300 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1, entries=50, sequenceid=566, filesize=7.1 K 2023-04-07 01:56:41,301 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=46.96 KB/48090 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 53ms, sequenceid=566, compaction requested=true 2023-04-07 01:56:41,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:41,301 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,301 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,302 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,302 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d93e024[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,304 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,306 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:41,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:41,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:41,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:41,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:41,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:41,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,307 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,312 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,312 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,315 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,315 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,317 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,317 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,318 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,320 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,321 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,321 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,321 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832601355, max timestamp=1680832601363Num uniques 859; 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832601355, max timestamp=1680832601363Num uniques 859; 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832601355, max timestamp=1680832601363Num uniques 859; 2023-04-07 01:56:41,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:41,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:41,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832661405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,421 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,424 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,424 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,425 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,425 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,426 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,426 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,428 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,434 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,434 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,435 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398c2f44[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,436 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,436 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,437 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,437 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,438 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,438 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,441 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,441 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,441 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,442 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,443 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,444 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,446 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,447 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,447 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,447 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,447 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 526 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832661508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,547 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,550 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,551 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,552 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,552 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,553 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,554 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,555 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,556 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507393c4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,558 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,559 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,559 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,562 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,564 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,564 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,564 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,565 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,566 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,567 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,572 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,572 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,573 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,573 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,573 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,673 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,676 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,676 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,677 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,679 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,679 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,680 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,681 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,681 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,682 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,682 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,683 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,683 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632292a[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,684 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,685 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,685 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,690 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,690 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,699 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,699 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,699 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,701 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,702 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,703 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,706 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,706 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,706 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,706 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,707 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,708 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:41,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 528 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832661711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,718 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407b25d8be5b08948efae5f5376a508e5e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b25d8be5b08948efae5f5376a508e5e5 2023-04-07 01:56:41,725 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=577, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7b83087444824536bcdf8adf98b22bb8 2023-04-07 01:56:41,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:41,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40144 deadline: 1680832661737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,741 DEBUG [Thread-4111] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:41,806 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,810 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,810 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,811 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,812 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,814 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,815 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,816 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,816 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,816 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,816 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,816 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,820 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,821 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a27b08[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,822 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,823 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,825 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,825 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,825 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,826 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,829 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,830 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,830 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,831 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,832 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,833 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,837 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,837 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,837 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:41,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,937 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:41,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:41,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:41,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:41,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:41,940 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:41,940 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:41,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:41,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,941 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,943 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,943 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,944 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,944 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:41,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:41,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:41,945 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,945 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:41,945 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:41,945 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:41,945 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:41,947 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:41,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0c82d7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:41,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:41,949 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,950 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:41,951 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,951 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:41,952 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,952 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:41,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:41,954 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:41,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:41,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:41,954 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:41,954 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:41,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,957 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,957 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:41,959 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:41,961 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:41,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:41,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:41,961 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:41,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:41,961 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:41,962 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:41,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:42,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 530 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832662014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,062 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,065 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,065 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,069 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,069 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,070 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:42,071 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,073 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,074 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,074 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,074 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102dd721[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,075 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,076 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,077 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,078 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,080 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,081 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,081 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,082 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,082 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,082 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,083 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,084 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,088 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,088 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,088 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,088 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/e3c574f847064ad984d8c729aa27b55e 2023-04-07 01:56:42,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/d4d6fad1d13c40ac9e8fc686facc8f19 2023-04-07 01:56:42,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7b83087444824536bcdf8adf98b22bb8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8 2023-04-07 01:56:42,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8, entries=150, sequenceid=577, filesize=25.4 K 2023-04-07 01:56:42,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/e3c574f847064ad984d8c729aa27b55e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e 2023-04-07 01:56:42,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e, entries=100, sequenceid=577, filesize=9.5 K 2023-04-07 01:56:42,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/d4d6fad1d13c40ac9e8fc686facc8f19 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19 2023-04-07 01:56:42,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19, entries=100, sequenceid=577, filesize=9.5 K 2023-04-07 01:56:42,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 852ms, sequenceid=577, compaction requested=true 2023-04-07 01:56:42,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:42,158 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:42,158 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 01:56:42,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:42,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:42,159 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97778 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:42,159 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:42,159 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,159 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42062 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 01:56:42,159 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/5ce4e59d32fe4a198b2e4c34bb9097f8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=95.5 K 2023-04-07 01:56:42,159 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:42,160 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,160 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/871798421f4c452b95ceaf4e4e8a8d80, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=41.1 K 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 5ce4e59d32fe4a198b2e4c34bb9097f8, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680832597969 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 871798421f4c452b95ceaf4e4e8a8d80, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680832597969 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting c42452d5ed81482c9da8a98d13135358, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832599106 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e0dab1d2f37b435f9e848ecd916952ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832599106 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 8f03660eac3e4e3dbc8bdf7fc0117d41, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680832601233 2023-04-07 01:56:42,160 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ad08a517343f4c75a6a1e59ad16078e5, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680832601247 2023-04-07 01:56:42,161 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7b83087444824536bcdf8adf98b22bb8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832601288 2023-04-07 01:56:42,161 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting e3c574f847064ad984d8c729aa27b55e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832601291 2023-04-07 01:56:42,169 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:42,185 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#630 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:42,188 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,190 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/dcf6d8dd382e4a75bc77d3686e0436b6 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/dcf6d8dd382e4a75bc77d3686e0436b6 2023-04-07 01:56:42,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,192 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,192 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,193 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,194 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,194 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into dcf6d8dd382e4a75bc77d3686e0436b6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:42,194 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,194 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,194 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=12, startTime=1680832602158; duration=0sec 2023-04-07 01:56:42,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:42,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:42,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,195 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 29910 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,196 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:42,196 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,196 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/79ec0d441e7645bcb8f48199f800c3b4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=29.2 K 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,196 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 01:56:42,196 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 79ec0d441e7645bcb8f48199f800c3b4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680832599106 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:42,196 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting b6a969a8ff00473f9e27cdbf08a664f1, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680832601247 2023-04-07 01:56:42,196 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:42,197 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:42,197 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting d4d6fad1d13c40ac9e8fc686facc8f19, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832601291 2023-04-07 01:56:42,208 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#632 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:42,220 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407cfa6da3f4f5040ae8afca31983344de6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfa6da3f4f5040ae8afca31983344de6 2023-04-07 01:56:42,222 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/7a5f5295aa6e4f84908cdb5764e6e374 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7a5f5295aa6e4f84908cdb5764e6e374 2023-04-07 01:56:42,225 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=614, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/92867b0cfd62463eac78e9d97401c431 2023-04-07 01:56:42,226 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into 7a5f5295aa6e4f84908cdb5764e6e374(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:42,226 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,226 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=13, startTime=1680832602158; duration=0sec 2023-04-07 01:56:42,227 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:42,235 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/ae8c9735566f4487bf5e95299d01aff1 2023-04-07 01:56:42,249 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/57f1ae306b2c44a4b81ceea66764ae21 2023-04-07 01:56:42,252 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/92867b0cfd62463eac78e9d97401c431 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431 2023-04-07 01:56:42,255 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431, entries=150, sequenceid=614, filesize=25.4 K 2023-04-07 01:56:42,256 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/ae8c9735566f4487bf5e95299d01aff1 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1 2023-04-07 01:56:42,259 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1, entries=150, sequenceid=614, filesize=11.9 K 2023-04-07 01:56:42,259 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/57f1ae306b2c44a4b81ceea66764ae21 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21 2023-04-07 01:56:42,262 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21, entries=150, sequenceid=614, filesize=11.9 K 2023-04-07 01:56:42,263 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 67ms, sequenceid=614, compaction requested=false 2023-04-07 01:56:42,263 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,263 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,263 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,263 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,263 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,263 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,263 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,265 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,265 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1c777[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,268 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,268 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,270 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,273 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,273 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,273 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,273 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,273 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,278 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,279 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,279 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,279 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,379 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,382 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,382 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,383 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,384 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,384 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,385 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,385 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,386 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,388 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,388 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,389 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,389 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac86bc[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,390 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,391 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,391 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,392 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,393 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,396 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,396 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,396 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,397 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,398 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,400 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,407 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,407 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,407 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,407 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,507 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,511 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,511 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,513 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,513 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,514 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,515 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,515 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,515 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,515 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,515 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,515 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,515 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,515 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,515 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,516 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,517 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288ea444[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,524 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,524 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,526 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,528 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,528 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,528 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,530 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,530 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,535 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,535 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,535 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,535 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:42,568 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:42,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:42,578 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040719edab1b094641cc9665e323a3021d8d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719edab1b094641cc9665e323a3021d8d 2023-04-07 01:56:42,602 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/b8c996bb66b4400eb67c806ad9395f8b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/b8c996bb66b4400eb67c806ad9395f8b 2023-04-07 01:56:42,606 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into b8c996bb66b4400eb67c806ad9395f8b(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:42,606 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:42,606 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=12, startTime=1680832602158; duration=0sec 2023-04-07 01:56:42,606 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832602610, max timestamp=1680832602619Num uniques 859; 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832602610, max timestamp=1680832602619Num uniques 859; 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832602610, max timestamp=1680832602619Num uniques 859; 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:42,635 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,638 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,638 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,639 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,641 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,641 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,642 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:42,643 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,644 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,645 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e4eaab[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,646 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,646 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,647 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,648 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,649 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,652 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,652 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,652 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,654 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,656 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,658 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,658 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,659 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,659 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,659 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,659 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,659 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:42,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 573 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832662665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,759 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,762 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,762 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,764 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,764 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,765 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,766 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,766 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,766 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:42,766 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,769 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,769 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,769 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,769 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f17757f[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:42,771 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 575 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832662768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,774 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,776 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,777 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,779 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,779 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,779 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,782 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,783 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,784 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,788 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,789 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,788 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,789 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,789 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,889 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:42,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:42,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:42,892 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:42,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:42,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:42,892 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,893 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,895 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,895 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:42,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:42,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:42,897 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,897 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:42,897 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:42,897 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:42,897 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:42,899 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:42,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfa9147[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:42,901 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:42,901 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,902 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:42,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:42,907 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:42,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:42,909 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:42,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:42,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:42,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:42,909 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:42,911 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,911 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:42,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,911 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:42,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,912 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:42,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:42,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:42,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:42,914 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:42,915 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:42,914 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:42,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:42,915 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:42,915 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:42,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:42,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 577 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832662973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:42,984 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=627, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/2f20e9c3ece742e8ab31008548310dee 2023-04-07 01:56:43,015 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:43,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:43,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:43,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:43,018 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:43,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:43,018 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:43,019 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,019 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,020 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,020 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:43,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:43,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:43,021 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,021 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:43,022 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:43,022 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:43,024 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,024 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:43,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760bacf8[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:43,026 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:43,026 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,027 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:43,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:43,027 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,027 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:43,027 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:43,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:43,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:43,031 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:43,031 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:43,031 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:43,031 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:43,031 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:43,032 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,032 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,032 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,033 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,034 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,037 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,037 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,037 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:43,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:43,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:43,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,038 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:43,038 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,038 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:43,138 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:43,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:43,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:43,142 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:43,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:43,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:43,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,142 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,143 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,143 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,144 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,144 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:43,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:43,146 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,146 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:43,146 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:43,146 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:43,147 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,148 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:43,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:43,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bb120[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:43,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:43,150 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,151 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:43,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:43,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:43,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,151 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:43,154 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:43,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:43,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:43,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:43,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:43,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:43,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:43,156 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:43,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,158 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,158 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,159 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,162 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:43,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:43,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:43,162 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,163 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:43,163 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,263 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:43,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:43,266 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:43,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:43,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:43,266 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,267 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,268 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,268 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:43,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:43,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:43,270 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,270 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:43,270 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:43,270 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:43,270 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:43,273 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6393803b[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:43,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:43,275 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,276 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:43,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,276 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:43,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 579 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832663278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:43,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:43,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:43,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:43,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:43,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:43,280 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:43,281 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,281 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,281 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,283 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:43,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,289 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:43,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,290 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:43,289 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,290 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,290 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,390 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:43,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:43,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:43,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/61cd279a4ec64c45b42b1db2b6de4445 2023-04-07 01:56:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:43,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:43,393 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:43,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:43,393 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:43,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,394 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,395 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,395 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,395 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,395 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:43,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:43,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:43,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,396 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:43,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:43,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:43,397 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:43,397 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:43,398 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,398 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,399 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:43,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9c55a7[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:43,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:43,401 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/436ac338354c4d31baaa503ff2fb735d 2023-04-07 01:56:43,402 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:43,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:43,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/2f20e9c3ece742e8ab31008548310dee as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee 2023-04-07 01:56:43,405 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:43,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,405 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:43,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:43,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee, entries=150, sequenceid=627, filesize=25.4 K 2023-04-07 01:56:43,408 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:43,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:43,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:43,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:43,408 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:43,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/61cd279a4ec64c45b42b1db2b6de4445 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445 2023-04-07 01:56:43,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:43,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:43,410 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,412 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445, entries=100, sequenceid=627, filesize=9.5 K 2023-04-07 01:56:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:43,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/436ac338354c4d31baaa503ff2fb735d as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d 2023-04-07 01:56:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:43,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:43,415 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:43,415 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:43,414 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:43,415 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:43,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:43,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:43,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d, entries=100, sequenceid=627, filesize=9.5 K 2023-04-07 01:56:43,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 848ms, sequenceid=627, compaction requested=true 2023-04-07 01:56:43,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:43,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:43,417 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:43,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:43,417 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:43,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:43,417 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:43,418 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:43,418 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:43,418 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,418 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:43,418 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/b8c996bb66b4400eb67c806ad9395f8b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=77.1 K 2023-04-07 01:56:43,418 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,418 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/dcf6d8dd382e4a75bc77d3686e0436b6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=34.1 K 2023-04-07 01:56:43,418 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting b8c996bb66b4400eb67c806ad9395f8b, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832599115 2023-04-07 01:56:43,418 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting dcf6d8dd382e4a75bc77d3686e0436b6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832599115 2023-04-07 01:56:43,419 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ae8c9735566f4487bf5e95299d01aff1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680832601391 2023-04-07 01:56:43,419 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 92867b0cfd62463eac78e9d97401c431, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680832601391 2023-04-07 01:56:43,419 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 61cd279a4ec64c45b42b1db2b6de4445, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832602565 2023-04-07 01:56:43,419 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 2f20e9c3ece742e8ab31008548310dee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832602560 2023-04-07 01:56:43,424 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:43,428 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#639 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:43,436 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/76e9765f4428481fbf8cb7935dab94e0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/76e9765f4428481fbf8cb7935dab94e0 2023-04-07 01:56:43,440 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into 76e9765f4428481fbf8cb7935dab94e0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:43,440 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:43,440 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=13, startTime=1680832603417; duration=0sec 2023-04-07 01:56:43,440 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:43,440 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:43,441 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:43,441 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:43,441 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,441 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7a5f5295aa6e4f84908cdb5764e6e374, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=34.1 K 2023-04-07 01:56:43,441 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 7a5f5295aa6e4f84908cdb5764e6e374, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680832599115 2023-04-07 01:56:43,442 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 57f1ae306b2c44a4b81ceea66764ae21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680832601391 2023-04-07 01:56:43,442 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 436ac338354c4d31baaa503ff2fb735d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832602565 2023-04-07 01:56:43,447 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#640 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:43,454 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/d29cda6273fb4cb19bf4ea4d59b0a5ec as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/d29cda6273fb4cb19bf4ea4d59b0a5ec 2023-04-07 01:56:43,458 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into d29cda6273fb4cb19bf4ea4d59b0a5ec(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:43,458 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:43,458 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=13, startTime=1680832603417; duration=0sec 2023-04-07 01:56:43,458 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:43,515 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:43,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:43,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:43,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:43,518 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:43,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:43,518 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:43,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,519 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,520 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:43,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:43,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,520 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:43,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:43,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:43,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:43,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:43,522 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:43,522 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:43,522 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:43,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:43,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:43,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 588 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832663796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,869 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/ad990ee5beef40d49f1ae4a72ffcec0e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/ad990ee5beef40d49f1ae4a72ffcec0e 2023-04-07 01:56:43,872 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into ad990ee5beef40d49f1ae4a72ffcec0e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:43,872 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:43,872 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=13, startTime=1680832603417; duration=0sec 2023-04-07 01:56:43,873 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:43,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 590 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832663899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:43,934 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304074f593ff2a26d45e39e1b6a1b377ebd36 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074f593ff2a26d45e39e1b6a1b377ebd36 2023-04-07 01:56:44,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 592 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832664102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,330 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-07 01:56:44,339 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=667, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7b0bca3fb2ac468aa53401797d7120ab 2023-04-07 01:56:44,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 594 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832664406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,748 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/3b50b1a2b00942a38ef7ddd6d03d7a1a 2023-04-07 01:56:44,756 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/03d68bf2165945c3aa1e794ea0f2987e 2023-04-07 01:56:44,759 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7b0bca3fb2ac468aa53401797d7120ab as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab 2023-04-07 01:56:44,762 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab, entries=150, sequenceid=667, filesize=25.4 K 2023-04-07 01:56:44,762 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/3b50b1a2b00942a38ef7ddd6d03d7a1a as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a 2023-04-07 01:56:44,765 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a, entries=150, sequenceid=667, filesize=11.9 K 2023-04-07 01:56:44,766 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/03d68bf2165945c3aa1e794ea0f2987e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e 2023-04-07 01:56:44,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40108 deadline: 1680832664768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,770 DEBUG [Thread-4115] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., hostname=jenkins-hbase4.apache.org,44961,1680832421898, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 01:56:44,771 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e, entries=150, sequenceid=667, filesize=11.9 K 2023-04-07 01:56:44,772 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1250ms, sequenceid=667, compaction requested=false 2023-04-07 01:56:44,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:44,772 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:44,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,774 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(9158): Flush requested on 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:44,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 01:56:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:44,774 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:44,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:44,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:44,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:44,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:44,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:44,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@375b56a0[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:44,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:44,777 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,778 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:44,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:44,778 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:44,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,778 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:44,779 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:44,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:44,780 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:44,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:44,783 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:44,783 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:44,783 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:44,783 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:44,783 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:44,785 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304074136faf2df834074b95481bd159fdf75 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074136faf2df834074b95481bd159fdf75 2023-04-07 01:56:44,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,785 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,786 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:44,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:44,786 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,787 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:44,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:44,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:44,790 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:44,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,790 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,791 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:44,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:44,791 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:44,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,791 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832604804, max timestamp=1680832604819Num uniques 859; 2023-04-07 01:56:44,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:44,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832604804, max timestamp=1680832604819Num uniques 859; 2023-04-07 01:56:44,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:44,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680832604804, max timestamp=1680832604820Num uniques 859; 2023-04-07 01:56:44,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 01:56:44,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832664849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832664849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,891 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:44,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:44,894 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:44,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:44,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:44,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:44,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,894 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,896 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:44,896 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:44,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,897 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:44,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:44,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:44,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:44,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:44,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:44,899 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:44,900 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:44,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc333da[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:44,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:44,902 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,904 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:44,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:44,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,904 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:44,904 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:44,906 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:44,906 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:44,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:44,909 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:44,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:44,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:44,909 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:44,909 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:44,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:44,910 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:44,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:44,911 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,913 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 596 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40064 deadline: 1680832664913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:44,919 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:44,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:44,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:44,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,919 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,919 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:44,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:44,919 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:44,920 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:44,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832664954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:44,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832664954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,019 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,022 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,022 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,023 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,023 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,023 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,024 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,024 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,025 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,026 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,026 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,026 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:45,026 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,028 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12aee401[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,029 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,030 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,031 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,031 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,031 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,035 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,037 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,038 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,038 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,039 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,039 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,039 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,039 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,040 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,043 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,043 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,043 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,043 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,143 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,147 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,147 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,148 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,148 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,149 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,150 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,150 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,150 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:45,150 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,153 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c61ba9e[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,156 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,157 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,157 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,157 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:45,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44961] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832665156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,159 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44961] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832665158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,161 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,161 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,161 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,163 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,164 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,165 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,169 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,169 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,168 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,168 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,169 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,169 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=679, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/889054b7ee454e768766ab91c0dc103b 2023-04-07 01:56:45,269 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,272 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,272 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,274 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,274 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,275 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,275 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,275 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:45,275 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,275 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,277 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e21fe4[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,278 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,280 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,280 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,284 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,286 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,286 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,287 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,287 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,287 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,288 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,289 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,291 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,293 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,293 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,293 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,293 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,393 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,397 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,397 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,402 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,402 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,403 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:45,404 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,406 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754bb1e1[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,408 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,409 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,409 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,410 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,410 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,411 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,411 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,413 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,413 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,413 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,415 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,416 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,417 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,419 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,420 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,420 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,420 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,420 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:45,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40136 deadline: 1680832665460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 01:56:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44961] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40068 deadline: 1680832665460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,520 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,524 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,524 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,525 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,525 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,526 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,527 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,528 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,528 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. as already flushing 2023-04-07 01:56:45,528 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,529 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a0dda5[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,532 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,534 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,534 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,536 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,538 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,538 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,538 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,540 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,540 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,540 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,541 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,542 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,544 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,544 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,545 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,544 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,545 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/10403eda69234be3b423d969aeea0160 2023-04-07 01:56:45,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/c976ed859b414f02a7f86f7ebf3aec93 2023-04-07 01:56:45,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/889054b7ee454e768766ab91c0dc103b as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b 2023-04-07 01:56:45,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b, entries=150, sequenceid=679, filesize=25.4 K 2023-04-07 01:56:45,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/10403eda69234be3b423d969aeea0160 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160 2023-04-07 01:56:45,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160, entries=150, sequenceid=679, filesize=11.9 K 2023-04-07 01:56:45,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/c976ed859b414f02a7f86f7ebf3aec93 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93 2023-04-07 01:56:45,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93, entries=150, sequenceid=679, filesize=11.9 K 2023-04-07 01:56:45,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 848ms, sequenceid=679, compaction requested=true 2023-04-07 01:56:45,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:45,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:45,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:45,623 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:45,623 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:45,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:45,623 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79024 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:45,623 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/A is initiating minor compaction (all files) 2023-04-07 01:56:45,624 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:45,624 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/A in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,624 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/B is initiating minor compaction (all files) 2023-04-07 01:56:45,624 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/d29cda6273fb4cb19bf4ea4d59b0a5ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=77.2 K 2023-04-07 01:56:45,624 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/B in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,624 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/76e9765f4428481fbf8cb7935dab94e0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=36.6 K 2023-04-07 01:56:45,624 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting d29cda6273fb4cb19bf4ea4d59b0a5ec, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832601404 2023-04-07 01:56:45,624 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 76e9765f4428481fbf8cb7935dab94e0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832601404 2023-04-07 01:56:45,625 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 7b0bca3fb2ac468aa53401797d7120ab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680832602652 2023-04-07 01:56:45,625 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 3b50b1a2b00942a38ef7ddd6d03d7a1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680832602652 2023-04-07 01:56:45,625 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] compactions.Compactor(207): Compacting 889054b7ee454e768766ab91c0dc103b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680832603789 2023-04-07 01:56:45,625 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 10403eda69234be3b423d969aeea0160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680832603789 2023-04-07 01:56:45,631 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#B#compaction#648 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:45,633 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#A#compaction#647 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:45,642 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/caaab92c3f264d07ac932d9bd62d9a19 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/caaab92c3f264d07ac932d9bd62d9a19 2023-04-07 01:56:45,645 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,646 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/B of 2c0ce65cbec62d633a29a3f2a06a7a49 into caaab92c3f264d07ac932d9bd62d9a19(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:45,646 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:45,646 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/B, priority=13, startTime=1680832605622; duration=0sec 2023-04-07 01:56:45,646 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 01:56:45,646 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 01:56:45,647 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 01:56:45,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,647 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1912): 2c0ce65cbec62d633a29a3f2a06a7a49/C is initiating minor compaction (all files) 2023-04-07 01:56:45,647 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c0ce65cbec62d633a29a3f2a06a7a49/C in TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,647 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/ad990ee5beef40d49f1ae4a72ffcec0e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93] into tmpdir=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp, totalSize=36.6 K 2023-04-07 01:56:45,648 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting ad990ee5beef40d49f1ae4a72ffcec0e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680832601404 2023-04-07 01:56:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,648 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting 03d68bf2165945c3aa1e794ea0f2987e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680832602652 2023-04-07 01:56:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,648 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,648 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,648 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] compactions.Compactor(207): Compacting c976ed859b414f02a7f86f7ebf3aec93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680832603789 2023-04-07 01:56:45,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,649 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,651 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/82d89af390b04bb8ba947bd05c6212f9 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/82d89af390b04bb8ba947bd05c6212f9 2023-04-07 01:56:45,653 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,653 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,654 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c0ce65cbec62d633a29a3f2a06a7a49#C#compaction#649 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 01:56:45,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,655 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,655 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/A of 2c0ce65cbec62d633a29a3f2a06a7a49 into 82d89af390b04bb8ba947bd05c6212f9(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:45,655 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:45,655 INFO [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/A, priority=13, startTime=1680832605622; duration=0sec 2023-04-07 01:56:45,655 DEBUG [RS:0;jenkins-hbase4:44961-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:45,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,656 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,656 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:45,656 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:45,657 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:45,657 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 01:56:45,671 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040758d5a99c580c4caf9c24606034158a6b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758d5a99c580c4caf9c24606034158a6b 2023-04-07 01:56:45,678 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=718, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/a8a492e31d70465ab700ef5550a7bde8 2023-04-07 01:56:45,685 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/36416d75ed7749ceba4dc1dd1b8b1445 2023-04-07 01:56:45,693 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/d140d5a873ff4830a1fb48711b7c8e37 2023-04-07 01:56:45,696 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/a8a492e31d70465ab700ef5550a7bde8 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/a8a492e31d70465ab700ef5550a7bde8 2023-04-07 01:56:45,698 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/a8a492e31d70465ab700ef5550a7bde8, entries=150, sequenceid=718, filesize=25.4 K 2023-04-07 01:56:45,699 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/36416d75ed7749ceba4dc1dd1b8b1445 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/36416d75ed7749ceba4dc1dd1b8b1445 2023-04-07 01:56:45,702 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/36416d75ed7749ceba4dc1dd1b8b1445, entries=150, sequenceid=718, filesize=11.9 K 2023-04-07 01:56:45,702 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/d140d5a873ff4830a1fb48711b7c8e37 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d140d5a873ff4830a1fb48711b7c8e37 2023-04-07 01:56:45,705 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d140d5a873ff4830a1fb48711b7c8e37, entries=150, sequenceid=718, filesize=11.9 K 2023-04-07 01:56:45,706 INFO [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 50ms, sequenceid=718, compaction requested=false 2023-04-07 01:56:45,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:45,706 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,708 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f42065[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,710 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,710 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,711 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,711 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,712 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,712 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,715 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,715 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,717 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,718 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,719 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,721 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,722 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,722 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,721 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,722 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,822 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 01:56:45,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 01:56:45,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 01:56:45,826 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 01:56:45,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 01:56:45,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 01:56:45,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,826 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,827 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,827 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,828 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,828 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 01:56:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 01:56:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 01:56:45,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 01:56:45,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 01:56:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 01:56:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 01:56:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 01:56:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 01:56:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 01:56:45,829 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. started... 2023-04-07 01:56:45,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:45,830 DEBUG [rs(jenkins-hbase4.apache.org,44961,1680832421898)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 01:56:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 01:56:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 01:56:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 01:56:45,835 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 01:56:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44961,1680832421898' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 01:56:45,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60781c80[Count = 0] remaining members to acquire global barrier 2023-04-07 01:56:45,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 01:56:45,837 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,838 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 01:56:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 01:56:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44961,1680832421898' in zk 2023-04-07 01:56:45,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,839 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 01:56:45,840 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 01:56:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,44961,1680832421898' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 01:56:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44961,1680832421898': 2023-04-07 01:56:45,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44961,1680832421898' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 01:56:45,842 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 01:56:45,842 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 01:56:45,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 01:56:45,843 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 01:56:45,843 INFO [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 01:56:45,844 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,844 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,844 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,844 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 01:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 01:56:45,846 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 01:56:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 01:56:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,849 DEBUG [(jenkins-hbase4.apache.org,34673,1680832420667)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 01:56:45,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 01:56:45,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 01:56:45,849 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 01:56:45,849 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 01:56:45,849 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 01:56:45,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 01:56:45,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 01:56:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 01:56:45,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 01:56:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 01:56:45,876 DEBUG [Thread-4128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:61971 2023-04-07 01:56:45,876 DEBUG [Thread-4128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,877 DEBUG [Thread-4124] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:61971 2023-04-07 01:56:45,877 DEBUG [Thread-4124] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,884 DEBUG [Thread-4122] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:61971 2023-04-07 01:56:45,884 DEBUG [Thread-4122] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,894 DEBUG [Thread-4120] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:61971 2023-04-07 01:56:45,894 DEBUG [Thread-4120] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,895 DEBUG [Thread-4126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:61971 2023-04-07 01:56:45,895 DEBUG [Thread-4126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,923 DEBUG [Thread-4117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:61971 2023-04-07 01:56:45,923 DEBUG [Thread-4117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,949 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 01:56:45,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 01:56:45,967 DEBUG [Thread-4113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:61971 2023-04-07 01:56:45,967 DEBUG [Thread-4113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:45,967 DEBUG [Thread-4109] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:61971 2023-04-07 01:56:45,967 DEBUG [Thread-4109] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:46,062 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/25a3858a453848b28edb825ad79f49a5 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/25a3858a453848b28edb825ad79f49a5 2023-04-07 01:56:46,066 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c0ce65cbec62d633a29a3f2a06a7a49/C of 2c0ce65cbec62d633a29a3f2a06a7a49 into 25a3858a453848b28edb825ad79f49a5(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 01:56:46,066 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:46,066 INFO [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49., storeName=2c0ce65cbec62d633a29a3f2a06a7a49/C, priority=13, startTime=1680832605623; duration=0sec 2023-04-07 01:56:46,066 DEBUG [RS:0;jenkins-hbase4:44961-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 01:56:51,818 DEBUG [Thread-4111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:61971 2023-04-07 01:56:51,818 DEBUG [Thread-4111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:54,866 DEBUG [Thread-4115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:61971 2023-04-07 01:56:54,866 DEBUG [Thread-4115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(394): wrote 477 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 01:56:54,866 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4625 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4661 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4412 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4647 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(398): read 4648 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:56:54,867 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50529bf3 to 127.0.0.1:61971 2023-04-07 01:56:54,867 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:54,867 INFO [Listener at localhost/45521] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 01:56:54,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 01:56:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:54,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:54,870 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832614870"}]},"ts":"1680832614870"} 2023-04-07 01:56:54,871 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 01:56:54,872 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 01:56:54,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, UNASSIGN}] 2023-04-07 01:56:54,874 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, UNASSIGN 2023-04-07 01:56:54,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:54,874 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832614874"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680832614874"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680832614874"}]},"ts":"1680832614874"} 2023-04-07 01:56:54,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898}] 2023-04-07 01:56:54,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:55,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:55,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2c0ce65cbec62d633a29a3f2a06a7a49, disabling compactions & flushes 2023-04-07 01:56:55,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. after waiting 0 ms 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:55,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2c0ce65cbec62d633a29a3f2a06a7a49 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=A 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=B 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c0ce65cbec62d633a29a3f2a06a7a49, store=C 2023-04-07 01:56:55,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 01:56:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:55,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304071445d6bb82154b35a79dd6a8b59e8de2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071445d6bb82154b35a79dd6a8b59e8de2 2023-04-07 01:56:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:55,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=728, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7fb5043b3a83449b87f9abf4d14328c4 2023-04-07 01:56:55,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/166a57ff504b4747864d455e62cfb1fb 2023-04-07 01:56:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:56,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/bba60b02679d40e5a4a288503a046944 2023-04-07 01:56:56,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/A/7fb5043b3a83449b87f9abf4d14328c4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7fb5043b3a83449b87f9abf4d14328c4 2023-04-07 01:56:56,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7fb5043b3a83449b87f9abf4d14328c4, entries=150, sequenceid=728, filesize=25.4 K 2023-04-07 01:56:56,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/B/166a57ff504b4747864d455e62cfb1fb as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/166a57ff504b4747864d455e62cfb1fb 2023-04-07 01:56:56,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/166a57ff504b4747864d455e62cfb1fb, entries=150, sequenceid=728, filesize=11.9 K 2023-04-07 01:56:56,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/.tmp/C/bba60b02679d40e5a4a288503a046944 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bba60b02679d40e5a4a288503a046944 2023-04-07 01:56:56,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bba60b02679d40e5a4a288503a046944, entries=150, sequenceid=728, filesize=11.9 K 2023-04-07 01:56:56,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 2c0ce65cbec62d633a29a3f2a06a7a49 in 1240ms, sequenceid=728, compaction requested=true 2023-04-07 01:56:56,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/9ba3dee1a9d742f4b1a3b56d0ed19117, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7f03a20be2644a808c5c2078d83aac3b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/47793098d3ba4954b9681e083e354632, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/5ce4e59d32fe4a198b2e4c34bb9097f8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/b8c996bb66b4400eb67c806ad9395f8b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/d29cda6273fb4cb19bf4ea4d59b0a5ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b] to archive 2023-04-07 01:56:56,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:56,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/f56f0a93b92c42de8ab7cc7f59e115ec 2023-04-07 01:56:56,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/adfe3af564324f0184f749376411e1e5 2023-04-07 01:56:56,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3e6e6ce3c1b840e38eaab8f27dbfe71b 2023-04-07 01:56:56,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/ba3a6f8808cd434dbfbe8db9cc8a4abe 2023-04-07 01:56:56,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/771642c4dcfe43639c2e5aaf7ccd5979 2023-04-07 01:56:56,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/e110ac17c0964cfd8b5ce2daba535ecf 2023-04-07 01:56:56,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/17d49993c7674ec3922b2c267d397c8d 2023-04-07 01:56:56,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/6953146e6ca3417aa8f3cc2a3494ec07 2023-04-07 01:56:56,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/9ba3dee1a9d742f4b1a3b56d0ed19117 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/9ba3dee1a9d742f4b1a3b56d0ed19117 2023-04-07 01:56:56,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/76ebd0a9e359432c8d517e067e216243 2023-04-07 01:56:56,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/bc3d365d4b7f4f6e959c64dde09f8ecf 2023-04-07 01:56:56,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7f03a20be2644a808c5c2078d83aac3b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7f03a20be2644a808c5c2078d83aac3b 2023-04-07 01:56:56,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/1b2dafbb15b9490795fffdabd1d11a98 2023-04-07 01:56:56,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/4caf3cc070194696af9e26ec9848ea01 2023-04-07 01:56:56,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/3a22193a17c5447ea9c53def85829efa 2023-04-07 01:56:56,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/47793098d3ba4954b9681e083e354632 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/47793098d3ba4954b9681e083e354632 2023-04-07 01:56:56,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/0ced0ea3aca847bcba1dbd4d95757e5a 2023-04-07 01:56:56,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/95dbc97d91314f689c35818b79e7f9c6 2023-04-07 01:56:56,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7c5c9f4feb124042b2a9b7c98130ac89 2023-04-07 01:56:56,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/5ce4e59d32fe4a198b2e4c34bb9097f8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/5ce4e59d32fe4a198b2e4c34bb9097f8 2023-04-07 01:56:56,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/c42452d5ed81482c9da8a98d13135358 2023-04-07 01:56:56,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/8f03660eac3e4e3dbc8bdf7fc0117d41 2023-04-07 01:56:56,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/b8c996bb66b4400eb67c806ad9395f8b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/b8c996bb66b4400eb67c806ad9395f8b 2023-04-07 01:56:56,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b83087444824536bcdf8adf98b22bb8 2023-04-07 01:56:56,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/92867b0cfd62463eac78e9d97401c431 2023-04-07 01:56:56,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/d29cda6273fb4cb19bf4ea4d59b0a5ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/d29cda6273fb4cb19bf4ea4d59b0a5ec 2023-04-07 01:56:56,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/2f20e9c3ece742e8ab31008548310dee 2023-04-07 01:56:56,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7b0bca3fb2ac468aa53401797d7120ab 2023-04-07 01:56:56,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/889054b7ee454e768766ab91c0dc103b 2023-04-07 01:56:56,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/276997731aca48f2a119890a4359a04e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/7a1d1a8f9329481b9485925bca5274c4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/5d04bf99404b466795bdafd30f79b1bd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/871798421f4c452b95ceaf4e4e8a8d80, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/dcf6d8dd382e4a75bc77d3686e0436b6, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/76e9765f4428481fbf8cb7935dab94e0, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160] to archive 2023-04-07 01:56:56,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:56,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/606615652abd41f4868e6229f4b3d2bf 2023-04-07 01:56:56,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0bb79ea0e34f47ba9cfc137ba4bff7c1 2023-04-07 01:56:56,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/c56c640e5c2a41eb926559a5ca8a84a8 2023-04-07 01:56:56,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/2082445c9e2b4e8b8b3af8b69ef898be 2023-04-07 01:56:56,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f56df9b05f36407693800fc0bc105fc1 2023-04-07 01:56:56,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/f0dfd88fbbef41e391f90024b4d11bf4 2023-04-07 01:56:56,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/32f89ee555c74f72a313349ffe14d232 2023-04-07 01:56:56,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/276997731aca48f2a119890a4359a04e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/276997731aca48f2a119890a4359a04e 2023-04-07 01:56:56,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/d10ba15b310a44fa8b70d9f156f57578 2023-04-07 01:56:56,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9b9dea9227fe4d01b0c1ecbfa315457b 2023-04-07 01:56:56,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/7a1d1a8f9329481b9485925bca5274c4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/7a1d1a8f9329481b9485925bca5274c4 2023-04-07 01:56:56,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/0d2096f228ae45918476722115e320e6 2023-04-07 01:56:56,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/39840401830b44fbbd64cf105f42bcbb 2023-04-07 01:56:56,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/80ea3de5dd83466cbcffd210bc5b218b 2023-04-07 01:56:56,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/eab9149636b741789bff7aa752784922 2023-04-07 01:56:56,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/5d04bf99404b466795bdafd30f79b1bd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/5d04bf99404b466795bdafd30f79b1bd 2023-04-07 01:56:56,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/525ca8e1adad4fb7b81efa4ff6b1cbe7 2023-04-07 01:56:56,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/64d2da08326c4a83af24d4a08accaaad 2023-04-07 01:56:56,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/871798421f4c452b95ceaf4e4e8a8d80 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/871798421f4c452b95ceaf4e4e8a8d80 2023-04-07 01:56:56,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/9f55b92999fd4893b06dee4300de57db 2023-04-07 01:56:56,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e0dab1d2f37b435f9e848ecd916952ad 2023-04-07 01:56:56,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ad08a517343f4c75a6a1e59ad16078e5 2023-04-07 01:56:56,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/dcf6d8dd382e4a75bc77d3686e0436b6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/dcf6d8dd382e4a75bc77d3686e0436b6 2023-04-07 01:56:56,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/e3c574f847064ad984d8c729aa27b55e 2023-04-07 01:56:56,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/ae8c9735566f4487bf5e95299d01aff1 2023-04-07 01:56:56,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/76e9765f4428481fbf8cb7935dab94e0 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/76e9765f4428481fbf8cb7935dab94e0 2023-04-07 01:56:56,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/61cd279a4ec64c45b42b1db2b6de4445 2023-04-07 01:56:56,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/3b50b1a2b00942a38ef7ddd6d03d7a1a 2023-04-07 01:56:56,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/10403eda69234be3b423d969aeea0160 2023-04-07 01:56:56,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/dc2c684b900a40f0aa3ca541580f3254, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a6b89dd6624942b498af76e9bf5878fc, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bbc70c232c9e4a40bc96d0c2864b8ad8, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/79ec0d441e7645bcb8f48199f800c3b4, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7a5f5295aa6e4f84908cdb5764e6e374, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/ad990ee5beef40d49f1ae4a72ffcec0e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93] to archive 2023-04-07 01:56:56,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 01:56:56,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a1c5eec9e52442fab626a196a525523e 2023-04-07 01:56:56,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/cb9c76d99b0d4361ad4199aa0b95c1ed 2023-04-07 01:56:56,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/9c11728ba84a4ef8be4c432633654097 2023-04-07 01:56:56,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/5422085766784ca6806a6b830d215c0e 2023-04-07 01:56:56,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/21a4797f6c02436aa55ee7f86125c8a2 2023-04-07 01:56:56,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/249ee2e39569471b818b39cca629c2ec 2023-04-07 01:56:56,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/911235177c274f59b1bc60a59633fe7a 2023-04-07 01:56:56,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/dc2c684b900a40f0aa3ca541580f3254 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/dc2c684b900a40f0aa3ca541580f3254 2023-04-07 01:56:56,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/4b1be77d852f47929b70573a0d2b3891 2023-04-07 01:56:56,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c744075e15444fd991d0c20225a5b1a7 2023-04-07 01:56:56,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a6b89dd6624942b498af76e9bf5878fc to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a6b89dd6624942b498af76e9bf5878fc 2023-04-07 01:56:56,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7516ab31423e4b8c809e584e1c066c80 2023-04-07 01:56:56,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/685c254da2cd4cc7b1eb4463ffe0bd1a 2023-04-07 01:56:56,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2450d32609314c5fb5e557cd13da5e8e 2023-04-07 01:56:56,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/8387993ba4cb46e0adfb9f828e273ebd 2023-04-07 01:56:56,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bbc70c232c9e4a40bc96d0c2864b8ad8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bbc70c232c9e4a40bc96d0c2864b8ad8 2023-04-07 01:56:56,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/09fef5c2ce324ded93cdfac8e948f63a 2023-04-07 01:56:56,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/2a121f01701b44b597c9bf9ee52602bf 2023-04-07 01:56:56,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/a9087f11f7a040e8afa54b4ded2ca955 2023-04-07 01:56:56,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/79ec0d441e7645bcb8f48199f800c3b4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/79ec0d441e7645bcb8f48199f800c3b4 2023-04-07 01:56:56,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/6354ca2c51984b44bcbd62025e64df22 2023-04-07 01:56:56,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/b6a969a8ff00473f9e27cdbf08a664f1 2023-04-07 01:56:56,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7a5f5295aa6e4f84908cdb5764e6e374 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/7a5f5295aa6e4f84908cdb5764e6e374 2023-04-07 01:56:56,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d4d6fad1d13c40ac9e8fc686facc8f19 2023-04-07 01:56:56,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/57f1ae306b2c44a4b81ceea66764ae21 2023-04-07 01:56:56,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/ad990ee5beef40d49f1ae4a72ffcec0e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/ad990ee5beef40d49f1ae4a72ffcec0e 2023-04-07 01:56:56,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/436ac338354c4d31baaa503ff2fb735d 2023-04-07 01:56:56,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/03d68bf2165945c3aa1e794ea0f2987e 2023-04-07 01:56:56,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/c976ed859b414f02a7f86f7ebf3aec93 2023-04-07 01:56:56,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits/731.seqid, newMaxSeqId=731, maxSeqId=4 2023-04-07 01:56:56,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49. 2023-04-07 01:56:56,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2c0ce65cbec62d633a29a3f2a06a7a49: 2023-04-07 01:56:56,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:56,372 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=2c0ce65cbec62d633a29a3f2a06a7a49, regionState=CLOSED 2023-04-07 01:56:56,372 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680832616372"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680832616372"}]},"ts":"1680832616372"} 2023-04-07 01:56:56,373 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-07 01:56:56,373 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 2c0ce65cbec62d633a29a3f2a06a7a49, server=jenkins-hbase4.apache.org,44961,1680832421898 in 1.4980 sec 2023-04-07 01:56:56,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-07 01:56:56,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c0ce65cbec62d633a29a3f2a06a7a49, UNASSIGN in 1.5000 sec 2023-04-07 01:56:56,375 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680832616374"}]},"ts":"1680832616374"} 2023-04-07 01:56:56,375 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 01:56:56,377 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 01:56:56,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5100 sec 2023-04-07 01:56:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 01:56:56,974 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-07 01:56:56,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 01:56:56,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:56,976 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-07 01:56:56,977 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:56,981 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:56,982 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C, FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits] 2023-04-07 01:56:56,984 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7fb5043b3a83449b87f9abf4d14328c4 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/7fb5043b3a83449b87f9abf4d14328c4 2023-04-07 01:56:56,985 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/82d89af390b04bb8ba947bd05c6212f9 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/82d89af390b04bb8ba947bd05c6212f9 2023-04-07 01:56:56,986 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/a8a492e31d70465ab700ef5550a7bde8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/A/a8a492e31d70465ab700ef5550a7bde8 2023-04-07 01:56:56,988 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/166a57ff504b4747864d455e62cfb1fb to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/166a57ff504b4747864d455e62cfb1fb 2023-04-07 01:56:56,989 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/36416d75ed7749ceba4dc1dd1b8b1445 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/36416d75ed7749ceba4dc1dd1b8b1445 2023-04-07 01:56:56,989 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/caaab92c3f264d07ac932d9bd62d9a19 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/B/caaab92c3f264d07ac932d9bd62d9a19 2023-04-07 01:56:56,991 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/25a3858a453848b28edb825ad79f49a5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/25a3858a453848b28edb825ad79f49a5 2023-04-07 01:56:56,992 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bba60b02679d40e5a4a288503a046944 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/bba60b02679d40e5a4a288503a046944 2023-04-07 01:56:56,993 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d140d5a873ff4830a1fb48711b7c8e37 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/C/d140d5a873ff4830a1fb48711b7c8e37 2023-04-07 01:56:56,994 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits/731.seqid to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49/recovered.edits/731.seqid 2023-04-07 01:56:56,995 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/.tmp/data/default/TestAcidGuarantees/2c0ce65cbec62d633a29a3f2a06a7a49 2023-04-07 01:56:56,995 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 01:56:56,995 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:56:56,996 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 01:56:56,997 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040713ba29c2308048c7b381930fe6927bcf to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040713ba29c2308048c7b381930fe6927bcf 2023-04-07 01:56:56,998 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071445d6bb82154b35a79dd6a8b59e8de2 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304071445d6bb82154b35a79dd6a8b59e8de2 2023-04-07 01:56:56,999 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719edab1b094641cc9665e323a3021d8d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040719edab1b094641cc9665e323a3021d8d 2023-04-07 01:56:56,999 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072cabab91894c46539335adc513a385f8 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072cabab91894c46539335adc513a385f8 2023-04-07 01:56:57,000 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e9264cd25694b26ac0faf658d480611 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072e9264cd25694b26ac0faf658d480611 2023-04-07 01:56:57,001 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040739472f17235d4717b0405306267f6225 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040739472f17235d4717b0405306267f6225 2023-04-07 01:56:57,001 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407399ef0b8d3784c63a0a793639231c038 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407399ef0b8d3784c63a0a793639231c038 2023-04-07 01:56:57,002 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074136faf2df834074b95481bd159fdf75 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074136faf2df834074b95481bd159fdf75 2023-04-07 01:56:57,003 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407428ece079e3949f59f9bdc4cf7d51a6d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407428ece079e3949f59f9bdc4cf7d51a6d 2023-04-07 01:56:57,003 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074f593ff2a26d45e39e1b6a1b377ebd36 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074f593ff2a26d45e39e1b6a1b377ebd36 2023-04-07 01:56:57,004 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757e9b38d940b430db63b1bf06662a083 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757e9b38d940b430db63b1bf06662a083 2023-04-07 01:56:57,005 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758d5a99c580c4caf9c24606034158a6b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758d5a99c580c4caf9c24606034158a6b 2023-04-07 01:56:57,005 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040766a756b0ad8944a394cd42a9f639fd22 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040766a756b0ad8944a394cd42a9f639fd22 2023-04-07 01:56:57,006 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076d732cd619ea4513836360b2c639e65b to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076d732cd619ea4513836360b2c639e65b 2023-04-07 01:56:57,007 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407768fe95a8aee46ada790b8048e385c2e to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407768fe95a8aee46ada790b8048e385c2e 2023-04-07 01:56:57,007 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078a234b481cf74845a26de4504fed731c to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078a234b481cf74845a26de4504fed731c 2023-04-07 01:56:57,008 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078c92d015cfa84e52b87611cd32766b7d to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304078c92d015cfa84e52b87611cd32766b7d 2023-04-07 01:56:57,009 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ab214f2dd63449e0ab11cb6d8af9a048 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ab214f2dd63449e0ab11cb6d8af9a048 2023-04-07 01:56:57,010 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b25d8be5b08948efae5f5376a508e5e5 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b25d8be5b08948efae5f5376a508e5e5 2023-04-07 01:56:57,011 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c1c3a94422664f4fa7f5a5232f6aa043 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c1c3a94422664f4fa7f5a5232f6aa043 2023-04-07 01:56:57,012 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c46166f218944894a54d51ec7efd74de to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407c46166f218944894a54d51ec7efd74de 2023-04-07 01:56:57,013 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cc031af133c34f90adbccca560151562 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cc031af133c34f90adbccca560151562 2023-04-07 01:56:57,014 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfa6da3f4f5040ae8afca31983344de6 to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407cfa6da3f4f5040ae8afca31983344de6 2023-04-07 01:56:57,015 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e48baa7ec70e4ed28fa37ba59911bbff to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e48baa7ec70e4ed28fa37ba59911bbff 2023-04-07 01:56:57,015 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eece8167f0e543fcb99abf16f680f86a to hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407eece8167f0e543fcb99abf16f680f86a 2023-04-07 01:56:57,016 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 01:56:57,017 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:57,019 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 01:56:57,020 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 01:56:57,021 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:57,021 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 01:56:57,021 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680832617021"}]},"ts":"9223372036854775807"} 2023-04-07 01:56:57,022 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 01:56:57,022 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2c0ce65cbec62d633a29a3f2a06a7a49, NAME => 'TestAcidGuarantees,,1680832582709.2c0ce65cbec62d633a29a3f2a06a7a49.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 01:56:57,022 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 01:56:57,022 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680832617022"}]},"ts":"9223372036854775807"} 2023-04-07 01:56:57,024 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 01:56:57,025 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 01:56:57,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 51 msec 2023-04-07 01:56:57,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-07 01:56:57,078 INFO [Listener at localhost/45521] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-07 01:56:57,089 INFO [Listener at localhost/45521] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=537 (was 549), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 359), ProcessCount=169 (was 169), AvailableMemoryMB=2600 (was 2670) 2023-04-07 01:56:57,089 INFO [Listener at localhost/45521] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-07 01:56:57,090 INFO [Listener at localhost/45521] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 01:56:57,090 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f3002be to 127.0.0.1:61971 2023-04-07 01:56:57,090 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:57,090 DEBUG [Listener at localhost/45521] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-07 01:56:57,090 DEBUG [Listener at localhost/45521] util.JVMClusterUtil(257): Found active master hash=419974866, stopped=false 2023-04-07 01:56:57,090 INFO [Listener at localhost/45521] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:56:57,091 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-07 01:56:57,091 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-07 01:56:57,091 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:56:57,091 INFO [Listener at localhost/45521] procedure2.ProcedureExecutor(629): Stopping 2023-04-07 01:56:57,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 01:56:57,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 01:56:57,092 DEBUG [Listener at localhost/45521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73bb993b to 127.0.0.1:61971 2023-04-07 01:56:57,092 DEBUG [Listener at localhost/45521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:57,092 INFO [Listener at localhost/45521] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44961,1680832421898' ***** 2023-04-07 01:56:57,092 INFO [Listener at localhost/45521] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-07 01:56:57,092 INFO [RS:0;jenkins-hbase4:44961] regionserver.HeapMemoryManager(220): Stopping 2023-04-07 01:56:57,093 INFO [RS:0;jenkins-hbase4:44961] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-07 01:56:57,093 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-07 01:56:57,093 INFO [RS:0;jenkins-hbase4:44961] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-07 01:56:57,093 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(3304): Received CLOSE for fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:56:57,094 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:57,094 DEBUG [RS:0;jenkins-hbase4:44961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x328b66e1 to 127.0.0.1:61971 2023-04-07 01:56:57,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fd36335cf6a580f3ebcc2199ad95d810, disabling compactions & flushes 2023-04-07 01:56:57,094 DEBUG [RS:0;jenkins-hbase4:44961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:57,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:56:57,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:56:57,094 INFO [RS:0;jenkins-hbase4:44961] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-07 01:56:57,095 INFO [RS:0;jenkins-hbase4:44961] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-07 01:56:57,095 INFO [RS:0;jenkins-hbase4:44961] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. after waiting 0 ms 2023-04-07 01:56:57,095 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:56:57,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fd36335cf6a580f3ebcc2199ad95d810 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-07 01:56:57,095 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-07 01:56:57,095 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1479): Online Regions={fd36335cf6a580f3ebcc2199ad95d810=hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810., 1588230740=hbase:meta,,1.1588230740} 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-07 01:56:57,095 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-07 01:56:57,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-07 01:56:57,096 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-07 01:56:57,097 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740, fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:56:57,151 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-07 01:56:57,297 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740, fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:56:57,309 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-07 01:56:57,309 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-07 01:56:57,497 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740, fd36335cf6a580f3ebcc2199ad95d810 2023-04-07 01:56:57,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/.tmp/info/f99ff7fe543a49bda0418e099b404dde 2023-04-07 01:56:57,506 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/info/28ca728f97c24490a51edb63fe59ee9e 2023-04-07 01:56:57,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/.tmp/info/f99ff7fe543a49bda0418e099b404dde as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/info/f99ff7fe543a49bda0418e099b404dde 2023-04-07 01:56:57,518 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 28ca728f97c24490a51edb63fe59ee9e 2023-04-07 01:56:57,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/info/f99ff7fe543a49bda0418e099b404dde, entries=2, sequenceid=6, filesize=4.8 K 2023-04-07 01:56:57,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for fd36335cf6a580f3ebcc2199ad95d810 in 424ms, sequenceid=6, compaction requested=false 2023-04-07 01:56:57,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/namespace/fd36335cf6a580f3ebcc2199ad95d810/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-07 01:56:57,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:56:57,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fd36335cf6a580f3ebcc2199ad95d810: 2023-04-07 01:56:57,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680832424926.fd36335cf6a580f3ebcc2199ad95d810. 2023-04-07 01:56:57,698 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 01:56:57,898 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 01:56:57,933 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/rep_barrier/f086a6bfd36a46cea2fdf32a72aee35f 2023-04-07 01:56:57,935 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f086a6bfd36a46cea2fdf32a72aee35f 2023-04-07 01:56:58,098 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-07 01:56:58,098 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-07 01:56:58,098 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 01:56:58,299 DEBUG [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 01:56:58,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/table/e96261a9d25e4ccab50335f40fe755b4 2023-04-07 01:56:58,344 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e96261a9d25e4ccab50335f40fe755b4 2023-04-07 01:56:58,344 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/info/28ca728f97c24490a51edb63fe59ee9e as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/info/28ca728f97c24490a51edb63fe59ee9e 2023-04-07 01:56:58,347 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 28ca728f97c24490a51edb63fe59ee9e 2023-04-07 01:56:58,347 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/info/28ca728f97c24490a51edb63fe59ee9e, entries=22, sequenceid=93, filesize=7.1 K 2023-04-07 01:56:58,348 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/rep_barrier/f086a6bfd36a46cea2fdf32a72aee35f as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/rep_barrier/f086a6bfd36a46cea2fdf32a72aee35f 2023-04-07 01:56:58,350 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f086a6bfd36a46cea2fdf32a72aee35f 2023-04-07 01:56:58,351 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/rep_barrier/f086a6bfd36a46cea2fdf32a72aee35f, entries=6, sequenceid=93, filesize=5.5 K 2023-04-07 01:56:58,351 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/.tmp/table/e96261a9d25e4ccab50335f40fe755b4 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/table/e96261a9d25e4ccab50335f40fe755b4 2023-04-07 01:56:58,353 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e96261a9d25e4ccab50335f40fe755b4 2023-04-07 01:56:58,353 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/table/e96261a9d25e4ccab50335f40fe755b4, entries=9, sequenceid=93, filesize=5.5 K 2023-04-07 01:56:58,354 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1259ms, sequenceid=93, compaction requested=false 2023-04-07 01:56:58,360 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-07 01:56:58,361 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-07 01:56:58,362 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-07 01:56:58,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-07 01:56:58,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-07 01:56:58,499 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44961,1680832421898; all regions closed. 2023-04-07 01:56:58,506 DEBUG [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/oldWALs 2023-04-07 01:56:58,506 INFO [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44961%2C1680832421898.meta:.meta(num 1680832424672) 2023-04-07 01:56:58,510 DEBUG [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/oldWALs 2023-04-07 01:56:58,510 INFO [RS:0;jenkins-hbase4:44961] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44961%2C1680832421898:(num 1680832424462) 2023-04-07 01:56:58,510 DEBUG [RS:0;jenkins-hbase4:44961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:58,510 INFO [RS:0;jenkins-hbase4:44961] regionserver.LeaseManager(133): Closed leases 2023-04-07 01:56:58,510 INFO [RS:0;jenkins-hbase4:44961] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-07 01:56:58,511 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-07 01:56:58,512 INFO [RS:0;jenkins-hbase4:44961] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44961 2023-04-07 01:56:58,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44961,1680832421898 2023-04-07 01:56:58,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 01:56:58,517 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@75136ccf rejected from java.util.concurrent.ThreadPoolExecutor@712cf6b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2564] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:58,517 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 01:56:58,517 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6bcbff01 rejected from java.util.concurrent.ThreadPoolExecutor@712cf6b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2564] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:58,519 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44961,1680832421898] 2023-04-07 01:56:58,520 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44961,1680832421898; numProcessing=1 2023-04-07 01:56:58,521 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44961,1680832421898 already deleted, retry=false 2023-04-07 01:56:58,521 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44961,1680832421898 expired; onlineServers=0 2023-04-07 01:56:58,521 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34673,1680832420667' ***** 2023-04-07 01:56:58,521 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-07 01:56:58,521 DEBUG [M:0;jenkins-hbase4:34673] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@583544e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-07 01:56:58,521 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34673,1680832420667 2023-04-07 01:56:58,521 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34673,1680832420667; all regions closed. 2023-04-07 01:56:58,522 DEBUG [M:0;jenkins-hbase4:34673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 01:56:58,522 DEBUG [M:0;jenkins-hbase4:34673] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-07 01:56:58,522 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-07 01:56:58,522 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680832424184] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680832424184,5,FailOnTimeoutGroup] 2023-04-07 01:56:58,522 DEBUG [M:0;jenkins-hbase4:34673] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-07 01:56:58,522 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680832424183] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680832424183,5,FailOnTimeoutGroup] 2023-04-07 01:56:58,522 INFO [M:0;jenkins-hbase4:34673] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-07 01:56:58,522 INFO [M:0;jenkins-hbase4:34673] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-07 01:56:58,522 INFO [M:0;jenkins-hbase4:34673] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-07 01:56:58,523 DEBUG [M:0;jenkins-hbase4:34673] master.HMaster(1512): Stopping service threads 2023-04-07 01:56:58,523 INFO [M:0;jenkins-hbase4:34673] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-07 01:56:58,523 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-07 01:56:58,523 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 01:56:58,523 ERROR [M:0;jenkins-hbase4:34673] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1981721381) connection to localhost/127.0.0.1:45891 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-07 01:56:58,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 01:56:58,523 INFO [M:0;jenkins-hbase4:34673] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-07 01:56:58,524 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-07 01:56:58,524 DEBUG [M:0;jenkins-hbase4:34673] zookeeper.ZKUtil(398): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-07 01:56:58,524 WARN [M:0;jenkins-hbase4:34673] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-07 01:56:58,524 INFO [M:0;jenkins-hbase4:34673] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-07 01:56:58,524 INFO [M:0;jenkins-hbase4:34673] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-07 01:56:58,525 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-07 01:56:58,525 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:56:58,525 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:56:58,525 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-07 01:56:58,525 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:56:58,525 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.88 KB heapSize=393.62 KB 2023-04-07 01:56:58,620 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 01:56:58,620 INFO [RS:0;jenkins-hbase4:44961] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44961,1680832421898; zookeeper connection closed. 2023-04-07 01:56:58,620 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@13df54f2 rejected from java.util.concurrent.ThreadPoolExecutor@712cf6b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2564] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:58,621 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): regionserver:44961-0x1015c5341aa0001, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 01:56:58,621 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@dbcac9f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@dbcac9f 2023-04-07 01:56:58,621 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@600657cd rejected from java.util.concurrent.ThreadPoolExecutor@712cf6b5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2564] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:58,621 INFO [Listener at localhost/45521] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-07 01:56:58,935 INFO [M:0;jenkins-hbase4:34673] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.88 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a516fe4344f4479a9888f3e22fb550e0 2023-04-07 01:56:58,939 INFO [M:0;jenkins-hbase4:34673] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a516fe4344f4479a9888f3e22fb550e0 2023-04-07 01:56:58,939 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a516fe4344f4479a9888f3e22fb550e0 as hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a516fe4344f4479a9888f3e22fb550e0 2023-04-07 01:56:58,942 INFO [M:0;jenkins-hbase4:34673] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a516fe4344f4479a9888f3e22fb550e0 2023-04-07 01:56:58,943 INFO [M:0;jenkins-hbase4:34673] regionserver.HStore(1080): Added hdfs://localhost:45891/user/jenkins/test-data/c3bb1703-fae1-1fd2-0095-09e85ed44874/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a516fe4344f4479a9888f3e22fb550e0, entries=65, sequenceid=578, filesize=24.7 K 2023-04-07 01:56:58,943 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegion(2948): Finished flush of dataSize ~344.88 KB/353156, heapSize ~393.60 KB/403048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 418ms, sequenceid=578, compaction requested=false 2023-04-07 01:56:58,944 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 01:56:58,944 DEBUG [M:0;jenkins-hbase4:34673] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 01:56:58,947 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-07 01:56:58,947 INFO [M:0;jenkins-hbase4:34673] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-07 01:56:58,947 INFO [M:0;jenkins-hbase4:34673] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34673 2023-04-07 01:56:58,962 DEBUG [M:0;jenkins-hbase4:34673] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34673,1680832420667 already deleted, retry=false 2023-04-07 01:56:59,064 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 01:56:59,064 INFO [M:0;jenkins-hbase4:34673] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34673,1680832420667; zookeeper connection closed. 2023-04-07 01:56:59,065 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64015780 rejected from java.util.concurrent.ThreadPoolExecutor@13968c2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4272] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:59,065 DEBUG [Listener at localhost/45521-EventThread] zookeeper.ZKWatcher(600): master:34673-0x1015c5341aa0000, quorum=127.0.0.1:61971, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 01:56:59,065 ERROR [Listener at localhost/45521-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4191f8d9 rejected from java.util.concurrent.ThreadPoolExecutor@13968c2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4272] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 01:56:59,066 WARN [Listener at localhost/45521] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-07 01:56:59,069 INFO [Listener at localhost/45521] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-07 01:56:59,178 WARN [BP-1148998398-172.31.14.131-1680832417299 heartbeating to localhost/127.0.0.1:45891] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-07 01:56:59,178 WARN [BP-1148998398-172.31.14.131-1680832417299 heartbeating to localhost/127.0.0.1:45891] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1148998398-172.31.14.131-1680832417299 (Datanode Uuid 46738926-ddcb-4327-bfe7-1a0a7e98fd71) service to localhost/127.0.0.1:45891 2023-04-07 01:56:59,182 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503/dfs/data/data1/current/BP-1148998398-172.31.14.131-1680832417299] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-07 01:56:59,183 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f9f25b5f-4b34-6594-6404-655fd0159b4d/cluster_9b76f59e-ae28-b994-34d1-d61176305503/dfs/data/data2/current/BP-1148998398-172.31.14.131-1680832417299] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-07 01:56:59,216 INFO [Listener at localhost/45521] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-07 01:56:59,389 INFO [Listener at localhost/45521] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-07 01:56:59,573 INFO [Listener at localhost/45521] hbase.HBaseTestingUtility(1293): Minicluster is down